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Abstrac  t 

Cost  effective  development  of  quality  software  for  new 
system  acquisitions  is  an  issue  of  increasing  concern  within 
the  Department  of  Defense  (DoD).  This  thesis  examines  the 
issue  of  software  development  for  Embedded  Computer  Systems 
<ECS) ,  within  the  context  of  the  Software  Development  Life 
Cycle  <SDLC),  from  the  perspectives  of  Software  Quality 
Assurance  (SQA)  and  Baseline  Management.  The  objective  of 
this  research  effort  is  to  develop  an  approach  for  the  joint 
application  of  SQA  and  Baseline  Management  to  improve 
management  control  [maintain  cost  and  schedule  integrity] 
during  the  software  development  process. 

Initially,  the  disciplines  of  SQA  and  Baseline  Manage¬ 
ment  are  presented  as  individual  components,  operating  dur¬ 
ing  the  SDLC,  which  provide  management  with  increased  con¬ 
trol  over  the  software  development  process.  General  con¬ 
cepts  associated  with  SQA  are  first  discussed,  including  the 
potential  role  of  Software  Metrics.  This  is  followed  by  a 
review  of  DoO  literature  pertaining  to  SQA  and  Baseline 
Management.  Having  explored  SQA  and  Baseline  Management 
individually,  SQA  and  Baseline  Management  are  then  studied 
as  a  combined  approach  towards  the  effective  management 
control  of  the  software  development  process. 


Through  the  use  of  a  structured  interview,  twenty-one 


Program  Managers  were  surveyed.  From  the  collected  cost, 
schedule  and  program  history  data,  programs  were  classified 
as  either  having  a  ‘Sound*  or  ‘Unsound*  SQA  program  and  as 
either  ‘adhering  to‘  or  ‘not  adhering  to*  a  baseline  manage¬ 
ment  standard.  Consequently,  analyzing  the  data  using  Stud¬ 
ent's  t  statistic,  the  major  finding  of  this  research  is 
that  there  is  no  statistical  evidence  that  the  application 
of  either  a  ‘Sound*  SQA  program  or  a  baseline  management 
standard  results  in  positive  cost  and  schedule  control. 


THE  EFFECTS  OF  SOFTWARE  QUALITY  CONTROL 
AND  BASELINE  MANAGEMENT  ON  THE 
ACQUISITION  OF  COMPUTER  PROGRAMS 


I .  I n  troduc  t i on 

Cost  effective  software  design  has  become  the  most  sig¬ 
nificant  development  problem  for  new  system  acquisitions. 
Over  the  four  generations  of  computer  systems  which  have 
evolved  in  the  past  three  decades,  computer  hardware  has 
been  reduced  to  a  miniaturized  scale,  achieved  very  high 
reliability  and  low  cost,  and  attained  virtually  infinite 
memory  capacity.  Computer  software  on  the  other  hand,  since 
it  is  new  being  asked  to  accomplish  very  complex  and  sophis¬ 
ticated  tasks,  has  become  less  reliable  and  significantly 
more  expensive.  Figure  1  typifies  the  inverse  cost  trend 
between  expenses  for  hardware  and  software.  Even  more  omi¬ 
nous  is  the  observation  that  three— four ths  of  the  annual 
expenditures  for  software  are  attributable  to  maintenance 
activities,  the  major  contributing  factor  being  design  (con¬ 
figuration)  changes  (47:2).  Within  the  Department  of 
Defense  (DoD),  the  increase  in  software  procurement  and 
maintenance  dollars  has  resulted  from  an  increased 
dependency  on  computer  software  for  weapons  system  opera¬ 
tion.  According  to  Dr.  Edith  U.  Martin,  Deputy  Under  Secre¬ 
tary  of  Defense  for  Research  and  Advanced  Technology, 

Virtually  every  system  in  the  current  and 
planned  inventory  makes  extensive  use  of  computer 
technology.  Computers  control  the  targeting  and 
flight  of  missiles,  coordinate  and  control  sophis- 
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ticated  systems  within  high  performance  aircraft, 
and  integrate  the  complex  activities  of  battle¬ 
field  command.  Consequently,  software  has  became 
the  dominant  factor  in  military  systems  (34:523. 

With  a  USAF  software  bill  now  exceeding  *2  billion  annually, 

and  with  increasing  dependency  on  software  for  national 

defense,  it  is  not  surprising  that  it  is  the  software 

element  of  the  computer  system  which  has  become  a  major 

object  of  DoD  concern. 

In  a  general  or  theoretical  sense,  software  development 
involves  a  series  of  interrelated,  time-phased  activities 
called  a  development  life-cycle  <44:2>.  The  Software 
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Figure  1.  Percentage  of  Software  and  Hardware  Costs 
of  Total  System  Acquisition  Costs. 
Source:  < 40:14). 


Development  Life  Cycle  consists  of  four  phases  (requirements 
analysis,  design,  coding  and  testing  and  integration);  soft¬ 
ware's  life-cycle  is  completed  in  a  fif  .*  phase  called 
operations  and  maintenance  (Figure  2).  The  requirements 
analysis  phase  is  that  phase  in  the  software  development 
life  cycle  that  the  role  which  the  software  system  is  to 
fulfill  is  defined.  In  the  design  phase,  software  design 
concepts  are  explored  in  an  attempt  to  satisfy  system  soft¬ 
ware  requirements.  The  coding  phase  is  the  phase  in  which 
the  actual  coding  is  accomplished.  Next,  in  the  test  and 
integration  phase,  the  developed  software  program  is  exe¬ 
cuted  to  uncover  problems  that  may  exist  in  the  code. 

Lastly,  the  operations  and  maintenance  phase  is  that  portion 


Figure  2.  The  Software  Development  and 
Operations  Life  Cycle. 
Source:  (29:3) 
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in  the  software  life  cycle  in  which  a  software  system  is 
turned  over  to  a  user  <22:67).  inserting  checkpoints  in 
this  development  process  enables  an  assessment  of  the  com¬ 
pleteness  and  adequacy  of  the  software  design  to  be  made 
early  in  project  development,  instead  of  later  in  the  pro¬ 
ject  when  recovery  may  be  impossible  without  adjustments  in 
schedule  and  cost.  The  software  development  life  cycle  is 
the  conceptualized  process  of  software  development  <44:240) 
and  software  quality  assurance  accomplishes  the  inspection 
of  this  process  <22:217). 

Management  of  software  acquisition  includes  those  acti 
vities  performed  to  develop  software  that  meets  performance 
requirements  while  maintaining  cost  and  schedule  integrity 
<6:70-71;  21:13;  23:7).  Further,  software  acquisition  man¬ 
agement  is  accomplished  through  the  collective  effort  of  a 
program  management  team  headed  by  a  Program  Manager  who  is 
solely  responsible  for  the  management  of  this  team  <1:20-1) 
To  make  efficient  and  effective  decisions,  the  program  mana 
ger  evaluates  information  solicited  from  the  other  team 
members  who  are  functional  specialists  in  such  areas  as 
quality  assurance,  configuration  management,  engineering, 
logistics,  and  program  control.  In  the  capacity  of  team 
leader,  the  program  manager  orchestrates  the  activities  of 
each  team  member  to  achieve  desired  results,  addresses  how 
well  these  results  satisfy  program  objectives,  and  melds 
these  individual  program  accomplishments  into  a  cohesive, 
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meaningful  collection  of  information.  While  administration 


of  each  discipline  is  vital  to  the  overall  program  success, 
the  emphasis  each  receives  is  dictated  by  the  nature  of  the 
system  being  acquired. 

Configuration  management,  however,  is  one  of  the  most 
important  parts  of  management  control  (12:5);  without  it, 
chaos  results  (37:45).  Configuration  management  deserves 
primary  emphasis  in  all  instances,  as  i t  wi 1 1  limit  or  en¬ 
hance  the  effectiveness  of  control  that  the  program  manager 
can  achieve  over  the  evolution  of  the  system's  design  (2:5). 
Configuration  Management  is  an  established  discipline  which, 
if  properly  implemented  and  kept  in  effect  throughout  the 
program,  will  help  the  program  manager. 

Department  of  Defense  Directive  (DoDD)  5010.1?  (15:2) 
defines  con-f  i  gurat  i  on  management  as  the  engineering  manage¬ 
ment  procedure  that  includes  the  following:  Configuration 
Identification;  Configuration  Control;  Configuration  Status 
Accounting;  and  Configuration  Audit.  Identification  is 
accomplished  by  a  process  known  as  baseline  management;  Con¬ 
figuration  Control  is  the  management  system  governing  changes 
made  to  an  established  baseline;  Status  Accounting  is  the 
process  which  provides  traceability  from  the  current  version 
of  an  item  or  its  related  documentation  to  its  original 
baseline;  and  Configuration  Audit  is  the  process  used  to 
cr.eck  an  item  for  compliance  with  the  configuration  identi¬ 
fication.  The  importance  of  the  configuration  management 
process  is  well  described  by  Alton  Patterson: 
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Con-figuration  management  is  perhaps  the  least 
understood,  and  most  difficult  to  enforce,  of  all 
disciplines  associated  with  the  management  and 
support  of  (Embedded  Computer  System}  software, 
yet  without  question,  it  is  one  of  the  most  impor¬ 
tant.  Many  technical  personnel,  and  even 
managers,  because  of  vague  understanding,  perceive 
configuration  management  as  a  time  consuming  over¬ 
kill  control  task  which  obstructs  the  software 
change  process.  Yet,  without  it,  programs  invai — 
iably  get  into  trouble  £41:283. 

While  all  four  elements  are  necessary  for  the  implementation 
of  a  sound  configuration  management  program,  this  thesis 
will  deal  exclusively  with  baseline  management  to  determine 
the  appropriate  timing  for  establishing  the  functional, 
allocated  and  product  baselines. 

AFR  65-3  defines  baseline  as:  A  configuration  identifi¬ 
cation  document,  or  a  set  of  such  documents,  formally  desig¬ 
nated  and  fixed  at  a  specific  time  during  a  Configuration 
Item's  life  cycle.  Baselines,  plus  approved  changes  for 
those  baselines,  constitute  the  current  approved  configura¬ 
tion  identification.  For  configuration  management,  there 
are  three  baselines,  as  follows: 

<a)  Functional  Baseline.  The  initial 
approved  functional  configuration  identification, 

(b)  Allocated  Baseline.  The  initial  approved 
allocated  configuration  identification,  <c)  Pro¬ 
duct  Baseline.  The  initial  approved  or  condition¬ 
ally  approved  product  configuration  identification 
£ 19:A-1 3 . 

Usually,  the  functional  baseline  is  documented  by  an  au¬ 
thenticated  system  specification.  The  system  specification 
establishes  the  general  performance  and  functional  require¬ 
ments  of  the  system.  The  allocated  baseline  is  documented 
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by  an  authenticated  development  specification.  Development 
specifications  'state  the  requirements  for  design  or  engi¬ 
neering  development  of  a  product  during  the  development 
period  [2:173.”  The  product  baseline  is  documented  by  an 
authenticated  product  specification  and  referenced  detail 
design  documents.  Product  Specifications  describe  the 
'"build  to"  form,  fit,  function,  and  interface  requirements 
and  the  acceptance  tests  for  these  requirements  [2:173.* 

In  this  thesis,  management  is  considered  to  be  the 
degree  of  effective  control  that  the  program  manager 
achieves  over  the  software  development.  It  is  measured  in 
terms  of  how  well  cost  and  schedule  integrity  is  maintained 
and  by  the  degree  of  quality  inherent  in  the  product.  Con¬ 
sequently,  while  configuration  management  can  aid  in  enhanc¬ 
ing  schedule  and  cost  integrity  during  the  software  develop¬ 
ment  effort,  developing  a  quality  software  product  is  also  a 
prime  concern,  a  concern  that  may  be  allayed  through  the 
incorporation  of  the  expanding  science  of  Software  Quality 
Assurance  (SQA). 

SQA  may  be  defined  as,  ”...  a  planned  and  systematic 
pattern  of  all  actions  necessary  to  provide  adequate  confi¬ 
dence  that  an  item  or  product  conforms  to  established  techni¬ 
cal  standards  143:3563.'  The  underlying  premise  is  that 
quality  is  a  property  of  software  which  is  designed  in 
rather  than  added  in  later  (10:185).  SQA  attempts  to  un¬ 
cover  problems  early  in  the  software  development  process 
and,  therefore,  significantly  reduce  the  probability  of 
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problems  appearing  later  in  the  program.  6ilb  concludes 
that  the  rule  seems  to  be  that  the  earlier  you  can  find  and 
fix  a  problem  in  a  software  system,  the  less  expensive  it 
will  be.  The  development  of  software  quality  assurance 
programs  appear  to  be  the  cost  effective  approach  to  reduc¬ 
ing  software  development  costs  and  improving  software  system 
qual i ty  <26:181). 

To  improve  the  quality  of  software,  the  consensus  in 
the  software  development  industry  is: 

The  quality  of  software  must  be  built  into 
the  software  during  the  software  development 
period.  This  implies  a  need  for  software  en¬ 
gineering  rather  than  the  partly  organized,  non- 
systematic  approaches  of  the  past.  Up  to  now  most 
technological  changes  have  occurred  in  hardware 
where  classical  quality  assurance  methods  applied, 
but  now  we  are  facing  a  conceptual  change  in 
products  122:6). 

Software  differs  radically  from  hardware  (22:8),  and  al¬ 
though  hardware  quality  assurance  practices  may  apply  to 
software  development,  in  all  probability,  traditional  hard¬ 
ware  quality  assurance  practices  must  adapt  to  the  nature  of 
software.  "Specifically,  the  concept  of  built-in  quality  or 
'doing  it  right  the  first  time'  will  have  to  be  emphasized 
122:83."  A  software  system  product  must  be  designed,  fabri¬ 
cated,  tested  and  documented  in  a  disciplined  fashion 
<44:174) . 

Control  is  the  essential  function  of  a  SQA  program. 
"Control  is  the  process  of  making  things  happen  in  conform¬ 
ance  with  established  standards.  The  basic  control  process 
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involves  establishing  standards,  measuring  performance 
against  these  standards,  and  correcting  any  deviations  from 
established  standards  [44:2053.”  Consequently,  SQA  is  an 
iterative  process  that  attempts  to  ensure  that  each  new 
phase  of  development  may  begin  only  if  the  preceding  phase 
of  work  has  been  performed  to  acceptable  standards.  How¬ 
ever,  for  a  SQA  program  to  be  effective,  it  must  be  planned 
and  executed  with  each  task  related  to  a  development 
ac  t  i  v  i  ty . 

According  to  Reifer  (44:264>,  the  objective  of  confi¬ 
guration  management  is  to  control  the  costs  and  the  reliabi¬ 
lity  of  a  software  system.  Further, 

.  .  .  if,  under  configuration  management, 
specific  system  requirements  were  identified  at 
the  start  of  a  software  development  effort;  then, 
as  the  effort  progressed  those  system  requirements 
could  be  checked  and  compared  to  determine  whether 
those  system  requirements  were  being  satisfied. 

Using  front  end  quality  planning  with  the  configu¬ 
ration  management  structure-  configuration  manage¬ 
ment  could  greatly  [improve]  the  software  manage¬ 
ment  effort  if  it  incorporated  a  definition  of 
quality  and  system  objectives  [29:173 

Consequently,  the  configuration  management  structure  can  be 
combined  with  SQA  methodologies  within  the  software  develop¬ 
ment  life  cycle  to  significantly  improve  DoD  procured  soft¬ 
ware  . 


Statement  of  Problem 

As  Gansler  states,  “the  most  critical  issue  facing  DoD 
is  the  increasing  use  of  and  dependency  on  software  in 
weapon  systems  without  the  proven  management  and  production 
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methods  necessary  to  control  its  direct  and  indirect  costs 
[27:1  l.“  As  the  complexity  and  cost  o-f,  and  dependence  on, 
computer  systems  continues  to  increase,  better  management  o-f 
computer  system  development  is  demanded  <3:ii).  Conse¬ 
quently,  the  question  becomes  one  o-f  how  can  we  better 
manage  the  development  o-f  so-ftware  to  ease  the  task  o-f  main¬ 
tenance  and  minimize  maintenance  attributed  to  program 
errors? 


Justification  o-f  the  Research 

The  work  to  be  accomplished  is  justified  by  the  major 
software  initiative  being  presently  heralded  in  the  DoD.  As 
Dr.  Edi th  U.  Martin  said: 

The  importance  of  software  to  DoD  systems  is 
obvious,  both  in  a  military  and  an  economic  sense. 

To  reap  the  benefits,  however,  DoD  software  must 
satisfy  certain  requirements. 

The  first  is  reliability.  First  and  fore¬ 
most  ,  we  must  ensure  that  our  systems  perform  the 
mission  for  which  they  are  intended,  particularly 
when  those  missions  include  life  critical  situa¬ 
tions — reliability  is  a  must.  Software  must  also 
be  adaptable.  We  must  be  able  to  react  quickly  to 
changing  missions  and  threats  by  easily  modifying 
software,  sometimes  within  hours,  perhaps  even 
minutes.  Finally,  software  must  be  affordable. 

DoD  is  concerned  about  the  cost  of  systems.  We 
must  find  ways  to  curtail  the  anticipated  burgeon¬ 
ing  cost  expected  as  part  of  the  DoD  software  bill 
134:531 . 


Objective  of  the  Research 

The  objective  of  this  research  effort  is  to  develop  a 
singular  approach  towards  the  application  of  SQA  and  base¬ 
line  management  to  effect  management  control  [maintain 
schedule  and  cost  integrityl  during  the  software  devel op- 


10 


ment  phase,  to  minimize  unnecessary  maintenance  require¬ 
ments  caused  by  design  errors  [achieving  reliability)  and 
to  ease  enhancement  tasks  during  the  operational  and  sup¬ 
port  phase  of  the  Embedded  Computer  System  (ECS)  life  cycle 
[designing  adaptable  and  flexible  software).  Consequently, 
this  research  effort  will  explore  how  the  discipline  of 
Baseline  Management  may  be  integrated  with  developing  SQA 
practices  and  associated  quality  assurance  tools  to  amelio¬ 
rate  the  management  of  software  development  and  quality  of 
delivered  software  products. 


Scope  of  the  Research 

Uhile  software  problems  experienced  in  the  DoD  apply 
to  both  Automated  Data  Processing  equipment  and  Embedded 
Computer  Systems,  this  thesis  is  only  concerned  with  the 
latter.  More  specifically,  this  thesis  addresses  software 
acquired  under  AFR-800  series  regulations.  Additionally, 
because  of  limited  time, the  research  will  only  focus  on 
the  software  development  life  cycle.  A  complete  study  will 
require  future  research  into  the  operations  and  maintenance 
phase.  However,  the  work  accomplished  in  the  effort  will 
serve  as  a  basis  for  such  future  work  and  will  provide 
important  empirical  insight  as  to  how  successfully  the 
government  is  currently  managing  software  development. 


irch  Questions 
The  following 


ch  questions,  compatible  with  the 


problem  statement  and  the  research  objectives,  guided  this 
research  effort* 


1.  Can  the  strict  application  of  configuration 
management,  specifically  baseline  management,  enhance 
the  management  effectiveness  of  software  development? 

2.  Can  the  application  of  existing  software  quality 
assurance  methodologies  be  efficiently  applied  to 
develop  a  higher  quality  software  product? 

3.  Can  a  configuration  management  program  and  a  soft¬ 
ware  quality  assurance  program  be  integrated  into  a 
single  plan  which  will  result  in  greater  benefits  than 
the  individual  application  of  either  of  these  methodo— 
1 ogi es? 


Plan  of  the  Report 

This  thesis  has  been  organized  into  six  chapters.  Here 


in  chapter  one,  the  reader  has  been  provided  a  general  ac¬ 
count  of  the  problematic  history  of  software  development. 

In  this  chapter,  it  was  also  postulated  that  the  use  of  SQA 
and  configuration  management  can  result  in  favorable  program 
outcomes  in  terms  of  program  cost,  schedule  integrity,  and 
end  product  quality.  Finally,  this  chapter  was  concluded 
with  a  problem  statement,  a  Justification  for  research,  an 
explanation  of  the  scope  of  research  and  research  questions. 

Chapters  two  and  three  are  1 i terature  reviews.  Chapter 
two  is  devoted  to  a  discussion  of  software  quality  assur¬ 


ance.  This  subject  is  being  presented  in  a  separate  chapter 


for  two  reasons.  First,  SQA  is  a  new  and  developing 
science.  Thus,  a  literature  review  would  provide  the  reader 
unfamiliar  with  SQA  with  a  basic  understanding  of  what  the 


experts  consider  to  be  the  substance  of  SQA.  Second,  the 
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literature  review  will  serve  as  a  -foundation  for  the 
development  of  an  information  gathering  instrument,  that  is, 
help  determine  what  quantitative  and  qualitative  data  must 
be  collected,  in  order  that  the  test  hypotheses  developed  in 
chapter  four  may  be  tested. 

Chapter  three  is  a  review  of  DoD  and  other  government 
documentation  which  provide  guidance  on  SQA  and  configura¬ 
tion  management.  This  review  was  completed  to  help  under¬ 
stand  what,  in  theory,  should  be  occurring  in  the  management 
of  developing  software.  This,  information  will  serve  as  a 
framework  for  comparing  what  should  be  happening  with  what 
is  happening  in  the  field. 

In  chapter  four,  the  methodology  used  to  develop  the 
survey  instrument,  the  structured  interview  process,  and  the 
statistics  to  be  utilized  within  this  thesis  effort  are 
presented.  Analysis  of  the  data  is  the  subject  of  chapter 
five.  Finally,  the  report  is  concluded  with  chapter  six  in 
which  the  results  are  examined.  Specifically,  an  evaluation 
is  made  of  how  well  the  objectives  of  the  thesis  are  met,  a 
discussion  of  what  the  authors  believe  the  findings  indi¬ 
cate,  and  finally  recommendat i ons  for  future  research  are 
made . 
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II 


.  Software  Qual  i  ty  Assurance 


kli£r  ature  Review 

On  April  10,  1981,  about  20  minutes  prior  to 
the  scheduled  launching  o-f  the  Space  Shuttle,  a 
software  -fault  in  the  real-time  control  computer 
system  forced  postponement  of  the  first  Shuttle 
orbital  flight.  Despite  thousands  of  hours  of 
testing  and  simulation,  this  hidde*-  fault  had  not 
been  detected  131:2483. 

This  is  only  one  example  of  many  documented  software 
failures  which  emphasizes  the  increasing  need  for  quality 
computer  programs.  Further,  recognizing  that  software  main¬ 
tenance  costs  vastly  outstrip  the  huge  initial  expenditures 
associated  with  software  development,  the  need  for  quality 
software  becomes  even  more  urgent.  With  this  understanding, 
this  chapter  will  focus  on  the  concepts  which  may  be  incorpo¬ 
rated  to  develop  quality  software.  To  begin  this  chapter, 
the  most  recent  1 i terature  concerning  SQA  discusses  SQA  as  a 
two  component  function,  with  the  first  component  relating  to 
software  quality  character i st i cs  and  the  second  component 
relating  to  Software  Quality  Management  practices. 

The  first  component  of  SQA  is  associated  with  soft¬ 
ware  quality  characteristics  or  attributes.  Poston  defines 
software  quality  as,  "The  totality  of  features  and  characte¬ 
ristics  of  a  software  product  that  bears  an  ability  to 
satisfy  a  given  need  143:3561."  Consequently,  software 
quality  may  be  viewed  as  concerned  with  those  characteris¬ 
tics  which  describe  the  degree  of  excellence  of  computer 
software  <10:7).  The  idea  of  software  possessing  character- 


i sties  o-f  quality  was  initially  explored  by  Ruby  and  Hart- 
wick  (45)  in  1968.  Ruby  and  Hartwick  identified  over  60 


candidate  software  character i st i cs.  In  more  recent  work, 
Walters,  Richards  and  McCall  (35)  have  reduced  proposed 
candidate  characteristics  to  a  set  of  11  software  charactei — 
istics,  which  are  contained  in  Table  I. 

TABLE  I 

Candidate  Quality  Character i st i cs 
Source:  (10:131) 


Portabi 1 i ty 
Reliability 
Correctness 
Ef  f i c i ency 
Integr i ty 
Usabi 1 i ty 
Mai ntai nabi 1 i ty 
Flexibility 
Testabi I i ty 
Reuseabi 1 i ty 
Interoperabi 1 i ty 


These  attributes  or  characteristics  are  understood  as 
imputing  software  quality.  For  example,  portability  is 
defined  as  the  property  which  allows  software  to  be  moved  to 
a  new  hardware  environment  with  relative  ease  (28:65). 
Specifically,  if  a  software  product  is  determined  to  possess 
the  characteristic  of  portability,  which  may  be  determined 
through  the  developing  science  of  Software  Metrics,  this 
implies  that  a  software  product  might  more  easily  and 
readily  be  adapted  to  new  uses,  which  is  considered  a 
desirable  quality  of  software.  Dunn  and  U1 lman  state  that, 
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“SQA  must  contribute  to  the  building-in  o-f  these  (Table  II 
attributes  in  the  development  of  software,  and  to  their 
retention  during  the  years  o-f  software  maintenance 
£22:2153 .” 

The  second  component  o-f  SQA  involves  the  role  of  moni¬ 
toring  adherence  to  standards.  And,  only  in  the  last  two  to 
-four  years  has  this  component  became  recognized  as  a  -formal 
requirement  (10:32).  Known  as  So-ftware  Quality  Management, 
“Software  Quality  Management  is  the  program  of  planned  and 
systematic  activities  to  determine,  achieve,  and  maintain 
computer  software  quality  £10:93.”  Software  Quality  Manage¬ 
ment  encompasses  the  broad  spectrum  of  management  activities 
undertaken  to  produce  quality  software.  Table  II  outlines  a 


TABLE  II 

A  Partial  List  of  Quality  Management  Activities 

Source:  (10:10) 


1.  Preparation  of  software  quality  management 
program  plan 

2.  Development  of  pol i c i es/procedures/standards 

3.  Software  quality  assurance  audits  of 
documentation,  design,  configuration  management, 
test i ng 

4.  Anal ysi s/eval uat i on/enforcement 

5.  Certif i cat i on/test i ng 

6.  Verification/validation 

7.  Education/training 

8.  Participation  in  design  reviews  and  configuration 
audi ts 

9.  Subcontractor  control 

10.  Preservat i on/handl i ng 

11.  Program  management  support 

12.  Identification  and  certification  of  tools, 
techniques,  and  methodologies 
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partial  list  of  Software  Quality  Management  activities.  Con¬ 
sequently,  software  quality  management  may  be  viewed  as 
i ncorporat i ng: 

1.  Planning.  A  Software  Quality  Management  plan 
would  serve  to  summarize  all  management  issues 
associated  with  software  development.  Management 
issues  would  include  such  items  as  the  assignment  of 
responsibilities,  timetables  for  milestone  events,  and 
testing  procedures.  Table  III  provides  a  sample  format 
for  a  Software  Quality  Management  plan. 

2.  Procedural  Assessments.  aSoftware  Quality 
Management  encompasses  assessment  of  the  procedures  and 
disciplines  used  in  the  development,  acquisition, 
management,  and  maintenance  of  the  software  product 
110:123 .* 

3.  Product  Assessment.  This  facet  of  software 
quality  management  concerns  the  review,  analysis, 
verification,  and  validation  of  the  software  product. 

Therefore,  Software  Quality  Management  is  concerned  with  the 

procedures  followed  in  the  development  of  software  and  makes 

up  the  second  component  of  the  SQA  function. 

Understanding  that  SQA  is  a  two  component  function,  an 

SQA  program  incorporates  and  fosters  the  development  of  both 

software  quality  characteristics  and  software  quality  manage 

ment  activities  in  an  attempt  to  develop  quality  software. 

The  difference  between  the  software  quality  characteristic 

component  and  the  software  quality  management  component  of 

the  SQA  function  is  in  the  type  of  control  exercised  in  the 

software  development  process.  Specifically,  software 

quality  characteristics  imply  control  through  the 

establishment  and  measurement  of  the  character i st i cs 

themselves;  software  quality  management  practices  imply 

control  through  the  specification  of  milestones  and  proce- 
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durt«  <10:11 >.  Consequently,  with  each  component  exercising 
control  over  a  different  facet  of  the  software  development 
effort  the  goal  of  designing  quality  into  the  software 
product  may  be  realized. 


TABLE  III 


Format  for  a  Software 
Quality  Management  Plan 
Source:  (10:12) 


1.0  Management  Overview 

1.1  Objectives  of  plan 

1.2  Schedule 

1.3  System  overview 

1.4  Management  control  procedures 

1.5  Organ i zat i on/resources 

2.0  System  Functional  Summary 

2.1  Information  required 

2.2  Software  development  process 

3.0  Software  Quality  Requirements 

3.1  Software  quality  factors 

3.2  Software  quality  metrics 

4.0  Life  Cycle  Tasks 

4.1  Software  quality  management  flowchart 

4.2  Task  descriptions 

5.0  Documentation  Requirements 


Software  Quality  Characteristics 

Wi th  regard  to  controlling  software  quality  charac¬ 
teristics  in  an  SQA  program,  *.  .  .  as  a  general  rule,  the 
system  designer  must  identify  all  essential  system  qualities 
[28:673.*  Furthering  this  premise.  Miller  and  Howden  have 
shown  that  the  degree  of  quality  a  person  puts  into  a 
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program  correlates  strongly  with  the  software  quality 
objectives  given  <38:29>.  Gilb  also  concludes  that  the  more 
clearly  a  goal  is  stated,  the  more  likely  it  is  that 
programmers  will  try  to  meet  that  goal  <28:67).  Therefore, 
from  the  literature  reviewed,  the  need  to  precisely  specify 
desirable  characteristics  within  software  is  essential. 

To  assist  in  this  task,  the  newly  developing  science  of 
Software  Metrics  is  providing  a  controlling  framework  within 
which  to  more  concisely  define  software  characteristics. 
"Software  metrics  is  a  new  area  of  science  aimed  at 
assigning  quantitative  indices  of  merit  to  software  [42:11." 
Software  metrics  provides  the  measurable  goal  against  which 
the  quality  of  software  can  be  assessed.  Metrics  provide 
quantifiable  measures  for  software  quality  by  measuring  the 
degree  to  which  a  software  product  possesses  and  exhibits  a 
certain  characteristic  <38:290).  In  Michael  Cooks  article, 
"Software  Metrics:  An  Introduction  and  Annotated 
Bibliography,"  the  concept  of  software  metrics  is 
introduced.  Software  metrics  is  viewed  as  a  developing 
science  which  will  provide  quantitative  measures  of  software 
characteristics  throughout  the  development  of  a  software 
product.  The  purpose  of  software  metrics  is  .  .  to  see 
how  they  reflect  the  quality  of  what  is  being  measured. 
Software  metrics  can  be  used  to  evaluate  the  effectiveness 
of  programming  methods  and  the  reliability  of  a  system 
[9:41 3."  In  concluding,  Cook  states,  "...  some  metrics 
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art  easy  to  apply  and  useful,  while  others  are  difficult  to 
implement  and  costly.  The  foundations  of  software  measure¬ 
ment  are  still  being  laid  and  software  metrics  will  be  an 
important  area  of  research  C9:433." 

While  software  metrics  presents  a  promising  opportunity 
to  significantly  improve  software  quality,  skepticism  exists 
in  the  scientific  community  concerning  the  employment  of  the 
scientific  method  in  the  development  of  metrics.  In  a 
recent  paper  by  Johnson,  the  significant  role  which  software 
metrics  could  fulfill  within  SQA  programs  is  recognized. 
However,  Johnson  is  skeptical  claiming  that  although  many 
metrics  have  been  developed,  ”...  many  have  yet  to  be 
demonstrated  and  validated  for  actual  use  130:1843.” 
Maintaining  a  similar  view,  Perl  is,  Sayward,  and  Shaw, 
editing  a  1981  publication  entitled  Software  Metrics, 
pose  the  question,  ”Can  there  be  assigned  to  software  and 
the  processes  associated  with  its  design,  development,  use, 
maintenance,  and  evolution,  indices  of  merit  that  can  sup¬ 
port  quantitative  comparisons  and  evaluation  of  software 
C42:pref aceJ?”  While  Dunn  and  Ul Iman  provide  an  excellent 
work  on  software  development  (22),  these  authors  strongly 
question  the  utility  of  software  metrics  (22:96).  Denicoff 
and  Grafton  summarize  the  challenge  put  forth  to  software 
metrics  researchers,  ”A  significant  challenge  to  software 
metrics  researchers  is  to  stay  within  the  traditional  para¬ 
digm  of  hypothesis,  evaluation,  criticism  and  review  114:2053.” 
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Despite  skepticism,  software  metrics  research  is  being 
pursued.  Murine  reports  promising  results  when  software 
metrics  were  included  in  a  SQA  program  on  a  major  de-fense 
system  so-ftware  project.  Murine  concludes,  "We  have  discov¬ 
ered  that  i ncorporat i on  o-f  the  So-ftware  Quality  Metrics 
(Software  Metrics!  methodology  into  an  SQA  program  has 
satis-fied  all  our  initial  quality  objectives  as  well  as 
some  not  previously  contemplated.  It  provides  a  real, 
positive  quality  impact  on  so-ftware  product  development  and 
measurement  and  can  be  used  as  a  major  cos  treduction  tool 
as  well  (39:188]." 

So-ftware  Quality  Management 

However,  it  should  be  recalled  that  so-ftware  attribute 
control  represents  only  hal-f  o-f  the  control  process  within 
an  SQA  program.  So-ftware  Quality  Management  practices  make 
up  the  remaining  hal-f  o-f  an  SQA  program.  Chow  summarizes 
the  basic  activities  or  practices  associated  with  so-ftware 
quality  management  concluding  that  so-ftware  quality 
management  consists  o-f  three  elements  (8:351 ): 

1 .  an  SQA  plan, 

2.  quality  control  techniques,  and 

3.  tools. 

Poston  expands  and  elaborates  on  Chow's  basic  elements  o-f 

so-ftware  quality  management.  Poston  concludes  that  a  soft 

ware  quality  management  program  includes  (43:354): 

1.  Policies.  A  set  of  policies  that  would  guide  the 
implementation  and  application  of  the  SQA  program. 
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2.  Methodologies.  This  refers  to  clearly  defining 
and  documenting  the  software  development  process 
throughout  the  software  development  life  cycle. 

3.  An  SQA  Plan.  This  plan  would  provide  a  writ¬ 
ten  record  of  all  SQA  activities  including  what 
part  of  the  organization  is  responsible  for  which 
activities. 

4.  Standards.  The  selection  and  creation  of 
standards  for  documents,  test  coverages,  configu¬ 
ration  management  reports  and  other  documentation 
as  outlined  in  the  SQA  plan. 

5.  Tools.  Tools  to  assist  in  the  software 
development  process  by  providing  the  mechanism  to 
ac-ccmplish  standards. 

6.  Reviews  and  Audits.  Provide  the  opportunity 
to  monitor  the  software  development  process. 

Poston's  elements  present  a  theoretical  framework  for  the 

development  of  a  software  quality  management  program  within 

which  software  quality  management  practices  may  be  applied. 

Further,  in  addition  to  the  above  mentioned  elements,  there 

exist  four  support  factors  which  also  contribute  to  enhance 

software  qual i ty. 

The  first  support  factor  which  contributes  to  the 
development  of  quality  software  is  management  support. 
Specifically,  full  support  of  management  must  exist  for 
software  quality  goals  to  be  realized  <10:64).  Dunn  and 
U1 lman  note  that  it  is  probably  a  mistake  for  management  to 
attempt  to  impose  a  complete  quality  system  all  at  once 
<22:253).  However,  a  formal  SQA  program  would  indicate  a 
commitment  by  management  to  the  requirements  for  a  quality 


operation,  in  addition  to  making  SQA  a  more  visible  and 
identifiable  function  <29: 7).  The  full  support  of  top 


management  is  viewed  as  the  -first  key  support  -factor  in  the 
development  o-f  quality  software. 

The  second  support  factor  derived  from  the  literature 
reviewed  implies  that  .  .  the  SQA  program  should  be  made 
an  independent  effort  by  a  functionally  separate  team 
[48:71."  Demarco  states  that  the  advantages  of  an  inde¬ 
pendent  team  include  <13:55): 

1.  estimators  having  no  emotional  stake  in  the 
project,  therefore,  estimators  are  relatively  free 
from  pressures  to  come  up  with  'the  right  answer,' 

2.  estimators  learning  through  substantial  repe¬ 
tition,  and, 

3.  a  centrally  controlled  data  collection  that 
will  result  in  homogeneous  measurement. 

Uhile  these  advantages  may  seem  obvious,  the  real  advantage 

is  the  management  visibility  provided  by  a  team  lacking  any 

conflict  of  interest  for  project  completion.  The 

establishment  of  an  independent  evaluation  team  is  the 

second  key  support  factor  which,  as  indicated  by  the 

literature  reviewed,  significantly  contributes  to 

"designing-in"  quality. 

The  third  support  factor  to  be  highlighted  relates  to 
the  testing  of  software.  Testing  implies  (44:211): 

1.  the  establishment  of  predetermined  goals  or 
standards, 

2.  the  measurement  of  performance,  and, 

3.  the  comparison  of  actual  performance  with 
standards. 

However,  while  the  implication  may  appear  to  be  that  of 
assessing  software  performance,  experts  conclude  that 
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.  .  the  primary  goal  of  tasting  should  not  be  tha  demon¬ 
stration  of  corract  performance  but  tha  axposura  of  hidden 
defects  [22:1803. ‘  This  perspective  is  substantiated  from 
previous  emphasis  on  the  fact  that  the  longer  a  defect  re¬ 
mains  in  the  system,  the  more  expensive  it  is  to  remove. 

The  testing  support  factor  is  operationalized  through  the 
use  of  software  testing  tools.  The  understanding  is  that 
" .  .  .  the  software  development  process  has  a  greater 
chance  of  success  if  the  instruments  for  measuring  the 
development  process  are  sufficiently  detailed  and  accurate 
and  used  at  a  sufficiently  early  point  in  the  work  process 
128:503."  Table  IV  contains  a  partial  list  of  software 
testing  tools.  Testing  is  the  third  key  support  factor  and 
is  viewed  in  the  1 i terature  as  imperative  in  the  software 
development  process. 

The  final  key  support  factor  identified  in  the  litera¬ 
ture  as  contributing  to  the  software  development  process 
relates  to  software  development  progression.  As  discussed 
previously,  configuration  management  establishes  a  series 
of  control  reviews.  With  respect  to  a  software  quality 
management  program,  work  should  only  be  allowed  to  progress 
when  all  specified  review  criteria  have  been  fulfilled 
<10:xiii;  22:21?) .  The  SQA  plan  must  decompose  the  whole 
software  development  process  into  stages.  At  the  end  of 
each  stage,  a  control  point  must  be  defined  with  a  complete 
specification  of  the  expected  product  and  the  quality 


criteria  -for  exiting  one  stage  and  entering  the  next  stage. 
Quality  assurance  methods  must  be  applied  at  each  control 
point  to  assess  the  quality  o-f  the  product  <8:351).  The 
idea  is  that,  ■ .  .  .  an  SQA  program  must  include  the 
provision  that  work  may  proceed  only  with  the  concurrence 
o-f  software  quality  assurance,  as  based  on  the  review  of 
previous  work  £22:2173.*  Insuring  software  progression 
criteria  are  met  combined  with  the  previously  mentioned  key 
support  factors  of  testing,  independent  inspection  and 
management  support  will  insure  a  higher  quality  software 
product . 


TABLE  IV 

Software  Testing  Tools 
Source:  <22:186) 


Basic  diagnostics 
Change  tracker 
Comparator 

Definition  and  design  processor 
Dynamic  analysis 
Emulator  and  simulator 
FI awchar ter 

Global  cross-ref erence  mapper 

Host  system 

Li brar i an 

Link  editor 

Postprocessors 

Preprocessors 

Regression  test  system 

Software  development  workbooks 

Standards  analyzer 

Static  analysis 

System  performance  monitor 

Test  case  generator 


Software  Quality  Management  serves  as  the  mechanism 
for  control  of  Software  Quality  Management  practices  within 
an  SQA  program.  In  an  analysis  o-f  the  literature  to  date, 
the  consensus  by  industry  experts  -focuses  on  an  effective 
software  quality  management  program  as  incorporating  such 
elements  as  previously  described  by  Poston  and  the  four  key 
support  factors  identified  above.  Software  Qual i ty  Manage¬ 
ment  is  the  second  component  of  an  SQA  program  and  further 
insures  that  quality  is  built  into  a  software  product  frem 
the  beginning  instead  of  attempting  to  add  it  at  the  end  of 
product  development  as  has  traditionally  been  the  case. 

Summary 

In  the  most  recent  literature  an  effective  SQA  program 
is  viewed  as  a  two  component  function.  The  first  component 
concerns  the  development  of  various  software 
characteristics,  which  are  defined  and  measured  within  the 
developing  science  of  Software  Metrics.  The  second 
component  concerns  various  Software  Quality  Management 
practices.  Consequently,  it  is  concluded  that  when  efforts 
are  made  to  "design-in*  quality  characteristics  within  a 
structured  framework  of  software  quality  management 
practices,  the  end  result  will  be  a  higher  quality  software 
product.  However,  given  that  the  basic  concepts  have  been 
defined  to  support  a  control  methodology  for  an  effective 
SQA  program,  an  anomaly  exists  in  that  reports  of  low 
quality  software  abound. 
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Continued  reports  of  high  software  development  costs 
and  low  user  satisfaction  with  procured  software  emphasizes 
the  need  for  developing  quality  software.  Consequently, 
this  chapter  has  focused  on  the  general  concepts 
contributing  to  the  development  of  quality  software  for  the 
two-fold  purpose  of,  first,  providing  the  reader  with  a 
basic  understanding  of  current  beliefs  and  methodologies 
purported  to  contribute  to  the  development  of  quality 
software  and,  second,  serving  as  the  foundation  for  a  data 
gathering  instrument,  which  will  eventually  be  used  to 
accept  or  reject  thesis  hypotheses.  In  the  next  chapter, 
DoO  SQA  will  be  reviewed,  together  with  baseline  management 
control  methodologies,  in  an  effort  to  discern  DoO 
practices  for  obtaining  quality  software. 
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III.  Government  Guidance  on  Con-figuration 

Management  and 
Software  Quality  Assurance 


Introduct i on 

Literature  devoted  to  so-ftware  management  control 
abound.  The  majority  o-f  these  expositions  can  be  divided 
into  two  categories.  The  -first,  because  o-f  the  exorbitant 
costs  being  incurred,  emphasizes  the  importance  o-f  e-f-fective 
management  and  control  during  the  development  o-f  the  so-ft- 
ware  product.  Such  material  was  used  to  develop  the  intro¬ 
ductory  chapter  and  is  -further  cited  throughout  this  report. 
The  second  category  addresses  the  di-f-ferent  approaches  or 
management  techniques  that  may  be  implemented  to  better 
manage  so-ftware  development  projects.  This  second  category 
is  the  emphasis  o-f  this  chapter. 

Basel ine  Management 

One  technique  which  is  repeatedly  identi-fied  as  a 
strong  management  control  tool  is  Configuration  Management. 
Only  a  modicum  of  the  literature,  however,  deals  with  the 
application  of  configuration  management  to  a  defined 
development  process.  DoD  directives,  and  other  related 
government  documents  are  perhaps  the  most  fertile  and  relev¬ 
ant  source  of  data  providing  such  direction.  As  Anway 
observes,  "...  the  directives  within  the  DoO  that  address 
configuration  management  are  written  for  the  acquisition 
process  of  major  defense  systems  13:13."  Table  V  lists  the 
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six  major  government  documents,  pertinent  to  Air  Force  ac¬ 
quisitions,  which  deal  configuration  management.  Examina¬ 
tion  of  these  documents  reveals,  however,  that  caution  must 
be  exercised  when  relying  on  these  directives,  as  ambigui¬ 
ties  and  contradictions  between  related  documents  risk  broad 
or  incorrect  interpretation.  Through  an  analysis  of  these 
six  documents,  the  remainder  of  this  section  will  include 
(1)  a  definition  of  baseline  management,  (2)  a  discussion  of 
document  precedence  within  the  Air  Force,  (3)  an  examination 
of  the  discrepancies  among  these  documents,  and  <4>  an 
indication  that  AFR  800-14  should  be  considered  the  proper 
guidance  document  for  acquisition  of  software  within  the 
Air  Force. 

TABLE  V 

Government  Documents  on  Configuration  Management 


DoDD  5010.1? 

AFR  65-3 
AFR  800-14 
MIL-STD— 483 
Ml L-STD-490 
M1L-STD-1521 

Air  Force  MIL-STD-483  describes  baseline  management  as 
"one  of  the  more  important  aspects  of  configuration  manage¬ 
ment  ...  which  is  formally  required  at  the  beginning  of 
an  acquisition  program  116:43.  MIL-STD-483  continues: 

Baselines  may  be  established  at  any  point  in 
a  program  where  it  is  necessary  to  define  a  formal 
departure  point  for  control  of  future  changes  in 
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performance  and  design.  System  program  management 
normally  employs  three  baselines  for  the  valida¬ 
tion  and  acquisition  of  systems  to  include  the 
functional,  allocated,  and  product  baselines.  .  . 

Baselines  are  the  basic  requirements  from 
which  contract  costs  are  determined  [16:53. 

DoD  Directive  (DoDD)  5010.1?  is  the  government  source 
document  which  establishes  top  level  guidance  for  configura¬ 
tion  management.  Uithin  the  Air  Force,  AFR  65-3,  Configura¬ 
tion  Management,  which  implements  DoDD  5010.19,  is  the 
authoritative  source  from  which  policy  on  timing  for  soft¬ 
ware  baseline  control  is  obtained.  According  to  AFR  65-3 
<19:2-1-2-2) : 


.  .  .  the  functional  baseline,  will  be  a 
product  of  the  conceptual  effort.  .  .  the  allo¬ 
cated  baseline,  will  be  formal W  established 
during  Advanced  Deve 1 opmen t/Va  ’at  ion  or  Full 
Scale  Development  .  .  .  the  product  baseline, 
shall  be  established  upon  successful  completion  of 
a  Physical  Configuration  Audit. 

Baseline  control  timing  is  the  same  in  AFR  800-14  (20:2-1-2-2) 
as  it  applies  this  policy  specifically  to  software  acquisi¬ 
tion.  The  authenticated  (baselined)  system  (functional) 
specification  is  the  definitive  document  resulting  from  the 
Conceptual  phase.  At  the  beginning  of  Full  Scale  Develop¬ 
ment  (FSD),  and  prior  to  Preliminary  Design  Review,  the 
development  specifications  should  be  completed  and  authenti¬ 
cated.  It  is  during  the  Production  phase  that  control  of 
the  product  baseline  is  established.  AFR  800-14  contains 
the  most  specific  software  policy  on  when  each  of  the  three 
identifying  baselines  should  be  established  and  is  explicit 
as  to  what  software  related  activities,  information  and 
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activities,  i  n  -forma  t  i  on  and  documentat  i  on  must  be  available 
before  the  government  takes  control . 

MIL-STD-483  also  attempts  to  establish  general  time 
-frames  for  appropriate  baseline  points  in  the  acquisition 
process.  According  the  MIL-STD-483: 

The  timing  of  the  establishment  of  the  func¬ 
tional  baseline  will  be  as  agreed  between  the 
contractor  and  the  procuring  activity,  but  not 
later  than  Preliminary  Design  Review  .  .  .  The 
allocated  baseline  will  be  formally  established 
with  the  award  of  engineering  or  operational  sys¬ 
tems  development  contract(s)  whenever  possible. 

The  timing  of  the  establishment  of  the  allocated 
baseline  will  be  as  agreed  between  the  contractor 
and  the  procuring  activity,  but  not  later  than  CDR 
[16:73. 

MIL-STD-483  does  not  specify  when  product  baseline  should  be 
establ i shed. 

In  contrast  to  MIL-STD-483,  which  is  concerned  specifi¬ 
cally  with  con  figuration  management  practices,  MIL-STD-1521 
outlines  conduct  for  the  different  design  reviews  and  audits. 
In  defining  the  purpose  and  objective  of  each  review  and 
audit,  it  provides  more  definitive  timing  for  bdu.eline  con¬ 
trol  actions.  According  to  MIL-STD-1521  (18:12,15,18,30), 
functional  baseline  should  be  taken  at  the  end  of  the  concep¬ 
tual  phase  or  beginning  of  the  validation  phase;  allocated 
baseline  should  be  established  well  before  Preliminary  Design 
Review  <PDR);  and  product  baseline  should  be  delayed  on  into 
the  product i on/operat i onal  phase  after  Physical  Configuration 
Audit  <PCA)  is  completed. 

APR  65-3  states  that,  aThe  configuration  management 
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process  shall  be  carefully  tailored  to  the  ...  nature  .  .  . 
of  the  Cl  involved  119:1-13."  The  intent  of  this  verbage 
has  often  been  mi  si nterpreted  and  resulted  in  varying  imple¬ 
mentation  from  one  program  to  another.  Uithin  the  Air  Force, 
it  should  be  understood  that  AFR  800-14  has  tailored  the 
process  for  ECS  and  should  be  the  standard  followed  for  a 
software  acquisition  program. 

Like  AFR  65-3,  M1L-STD-1 521 ' s  authority  is  also  jeopard¬ 
ized.  The  source  is  from  a  'note"  within  that  document  which 
reads:  "Actual  time  phasing  of  activities  must  be  tailored  to 
each  program  118:103."  MIL-STD-483  is  the  most  guilty  of  the 
discussed  sources  which  frustrates  the  issue  of  when  to 
establish  the  three  baselines.  As  may  have  been  noted  from 
above,  MIL-STD-483  allows  functional  baseline  to  be  delayed 
up  through  PDR  and  allocated  baseline  to  be  established  as 
late  as  Critical  Design  Review.  Additionally,  another  source 
of  weakness  in  MIL-STD-483,  like  MI L-STD-1 521 ,  stems  from  a" 
note"  which  reads: 

On  some  programs,  particularly  major  pro¬ 
grams,  forcing  establishment  of  allocated  baseline 
as  early  as  90  days  after  award  artificially  re¬ 
stricts  design  solutions  causing  costly  changes 
downstream.  The  agreement  should  be  documented 
e.g.,  in  the  configuration  management  plan,  if  one 
is  used  on  the  program/projec t  116:73. 

The  flexibility  built  into  these  documents  are  perhaps  work¬ 
arounds  made  available  to  account  for  the  many  cir.u«,.9tances 
that  exist  in  the  acquisition  of  a  product.  However,  as  the 
literature  indicates,  there  exists  a  natural  order  of  events 
which  occurs  in  all  acquisition  programs.  Theref ore ,  a 
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program  manager  should  not  be  willing  to  proceed  from  one 
event  to  the  next  without  being  satisfied  that  required 
documentation  is  available  and  related  activities  are  accomp 
1 i shed. 

It  is  true  that  a  program  may  not  have  had  Conceptual 
or  Validation  phase.  However,  even  if  its  inception  is  at 
the  beginning  of  FSD,  a  top  level  < system/pr ime  item)  specifica¬ 
tion  i s  st i 1 1  needed;  development  specifications  are  based 
upon  the  allocation  of  requ i rements  from  the  top  level 
specifications;  and  product  specifications  are  a  complement 
to  their  associated  development  specification.  This  sequence, 
and  its  relations  to  program  engineering  design  reviews,  is 
set  in  AFR  800-14,  which  governs  the  acquisition  of  embedded 
computer  resources.  Therefore,  determination  for  baseline 
control  subsequently  used  in  the  remainder  of  this  thesis 
shall  be  based  on  the  discussions  set  forth  in  AFR  800-14. 

Software  Quality  Assurance 

Uithin  the  OoD,  quality  assurance  is  viewed  as  the, 

'.  .  .  planned  and  systematic  pattern  of  all  actions  neces¬ 
sary  to  provide  adequate  confidence  that  material,  data, 
supplies  and  services  conform  to  established  technical  re¬ 
quirements  and  achieve  satisfactory  performance  117:13.* 

With  reference  to  SQA,  Table  VI  outlines  the  DoD  documents 
which  reference  software  quality. 

DoD  initiatives  to  improve  the  quality  of  software 
obtained  began  in  the  mid  l?70's.  Prompted  by  problems 
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which  still  exist  in  the  software  industry  today,  the 
government  began  a  review  o-f  how  systems  were  acquired. 

Known  as  the  Acquisition  Improvement  Program,  the  -focus  was 
on  increased  acquisition  effectiveness  and  efficiency.  As  a 
result  o-f  these  efforts  to  improve  the  Acquisition  process, 
DoDD  5000.1  was  published  (32:1).  The  policy  and  management 
principles  applicable  to  major  system  acquisitions  are 
established  in  this  directive.  Specifically,  DoDD  5000.1 
outlines  the  responsibilities  of  top  level  managers  and  key 
decision  points  in  the  system  acquisition  process.  However, 
aDoDO  5000.1  doesn't  address  computer  resources  (hardware, 
software,  personnel,  facilities,  etc.>  or  software 
specifically  .  .  .  £32:11."  Consequently,  while  DoDD  5000.1 
was  the  initial  attempt  to  improve  system  acquisition,  it 
was  left  to  later  documents  to  recognize  the  significance  of 
software  in  the  system's  acquisition  process. 

One  of  the  first  top  level  DoD  documents  to  specifi¬ 
cally  recognize  the  significance  associated  with  software  is 
DoDD  5000.2.  This  directive  emphasizes  that  plans  for  soft¬ 
ware  development,  documentation,  testing  and  updating  during 
software  development  require  special  attention  (32:17>. 

DoDD  5000.2  establishes  the  policies  and  procedures,  in¬ 
cluding  policy  guidance  for  the  acquisition  of  computer  re¬ 
sources,  for  DoD  activities  in  support  of  DoDD  5000.1. 

DoDD  5000.2?  is  also  an  upper  level  management  document 
which  should  be  considered.  This  document  establishes  the 
DoD  pol icy  for  the  management  and  control  of  computer  re- 
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TABLE  VI 


DoD  Software  Quality  Documents 
Source:  <10:102) 


OMB  A-109/0FPP  P**l.l 


1  i 

DSAH  8200.1  MIL-Q-9858A 

' 

DODD 

5000 . 1 

1 

DEF  SYST 

NATO 

DOD-STD-SQA 

DODD 

5000.2 

SMP 

SU 

A  QAP 

DOD- STD-480 A  ! 

DODD 

5000.3 

SU  R&D 

JPCG- 

! 

TECH 

PLAN 

CRM 

MI L-S— 83490 

DODD 

5000.29 

ECR/ 

JOINT 

j 

l 

j 

DSARC 

GUIDE¬ 

BOOK 

CM  REG 

MI L-STD-490  j 

DODD 

4155.1 

MI L— STD— 881 A  i 

DODD 

4120.21 

ARMY 


AR  18-1 

MIL-S-52779  <AD) 


NAVY 


MI L-STD-1 67?  (NAVY) 
SQA  PLAN  DID 
TADSTAND  9 


AIR  FORCE 


AFR  800-14 
AFSCR  74-1 
MIL- STD-4 83  OJSAF) 
SQA  GUIDEBOOKS 
CPDP  DID 
QAP  DID 


sources  with  increased  emphasis  on  software.  DoDD  5000.29 
establishes  policy  for  the  management  and  control  of  compu¬ 
ter  resources  during  the  development,  acquisition,  deploy¬ 
ment  and  support  of  major  defense  systems.  The  provisions 
of  this  document  encompass  major  programs  as  designa  ted  by 
DoDD  5000.1,  but  its  principles  are  also  to  be  applied  in 
the  acquisition  of  defense  systems  that  do  not  fall  in  the 
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' major  acquisition  category''  121:273.  In  addition,  OoOO 
5000.29  specifies  milestones  and  the  need  to  associate  spe¬ 
cific  criteria  to  determine  milestone  attainment  through  the 
Software  Development  Life  Cycle.  DoDD  5000.29  provides 
policy  guidance  for  the  management  of  computer  resources. 

The  various  5000  series  directives  discussed  in  this 
paper  indicate  an  upper  level  <DoD)  managerial  commitment  to 
developing  quality  software;  the  Air  Force  800  series  regu¬ 
lations  implement  the  5000  series  policy.  Specifically,  AFR 
800-14  deals  with  the  acquisition  and  support  for  computer 
resources  in  weapon  systems;  it  is  the  implementation  of 
DoD  directives  5000.1,  5000.2  and  5000.29.  AFR  800-14  is 
the  first  in  the  series  of  Air  Force  <AF>  acquisition  or 
management-oriented  regulations  to  specifically  address 
software  <21:28>.  "The  primary  A F  policy  on  the  management 
of  computer  re  lurces  in  weapon  systems  is  contained  in  AFR 
800-14  C  32 • 1 1 . “ 

AFR  800-14  is  a  two  volume  publication.  The  first 
volume  of  AFR  800-14  focuses  on  top  level  functional  manage¬ 
ment  support  for  computer  resources  and  systems.  The  pur¬ 
pose  of  this  volume  is  stated  as: 

.  .  .  insure  that  computer  resources  in  sys¬ 
tems  e* e  planned,  developed,  acquired,  employed, 
and  supported  to  effectively,  efficiently  and 
economically  accomplish  Air  Force  assigned  mis¬ 
sions  [20:33. 

This  volume  emphasizes  the  importance  of  the  correct  defini¬ 
tion  of  software  requirements. 


is  meant  to  be 


The  second  volume  of  AFR  800-14,  .  . 

a  definitive  treatment  of  the  policies,  procedures  and  guid¬ 
ance  required  to  manage  computer  resource  development,  as 
defined  by  AFR  800-14,  Vol  I  132:573."  This  volume  covers  a 
broad  spectrum  of  management  topics  from  the  planning  and 
engineering  of  computer  resources  to  documentation,  testing 
and  configuration  management  as  related  to  computer  re¬ 
sources.  In  addition,  AFR  800-14  is  of  special  interest 
since  this  regulation  serves  as  the  source  document  for  the 
establishment  of  an  SGA  program  during  software  development. 
Specifically,  AFR  800-14  outlines  the  requirement  for  a 
Computer  Program  Development  Plan  (CPDP).  The  CPDP  states 
the  steps  required  to  complete  a  project  and  the  methods 
that  will  be  followed  in  each  step  <22:21<S).  Specifically, 
the  CPDP  is  concerned  with  the  controls  required  to  generate 
quality  software. 

The  CPDP  is  considered  to  be  a  management 
plan  and,  hence,  a  living,  adaptable  document 
which  defines  objectives,  breaks  down  work  tasks, 
assigns  work  schedules  and  institutes  the  monitor¬ 
ing  required  to  feed  back  progress  estimates 
132:83 . 

It  is  of  interest  to  note  that  AFR  800-14  does  not  specifi¬ 
cally  address  SQA  but  instead  leaves  SQA  to  be  dealt  with 
through  the  CPDP  and  supporting  Military  Standards.  AFR 
800-14  is  concerned  with  how  the  computer  resource  acquisi¬ 
tion  process  should  be  implemented. 

In  support  of  AFR  800-14,  there  exist  several  documents 
which  are  often  imposed  on  contractual  requirements.  MIL- 
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STD-483  establishes  uni-form  con-figuration  management  prac¬ 
tices  and  spec i -f i cat i ons  which  can  be  applied  to  all  USAF 
systems,  equipment,  munitions  and  computer  programs.  MIL- 
STD-490  describes,  ■ the  purposes,  formats  and  technical  con¬ 
tent  of  various  types  of  specifications  ...  £ 32:493. ■ 

And,  I1IL-STD-1521A  describes  the  requirements  for  conducting 
the  various  milestone  events.  "This  standard  outlines  the 
minimum  information  required  for  each  type  of  review  and 
audit  £25:1773.*  Although  these  documents  do  not 
specifically  detail  SQA  programs,  the  need  for  SQA  is  recog¬ 
nized  through  requirements  for  testing.  One  criticism 
levied  on  MIL-STDs  483  and  490  is  that,  while  these  docu¬ 
ments  do  indicate  milestones  and  performance  testing,  the 
actual  work  to  be  accomplished  and  products  to  be  delivered 
are  not  defined  <21:25).  However,  according  to  Driscoll, 
MIL-STD-1521A  does  define  work  and  product  deliverables. 

A  more  recent  standard  which  deals  exclusively  with  SQA 
is  MIL-S-52779A,  "Software  Quality  Assurance  Program  Require 
ments."  *MIL-S-52779A  establishes  the  requirement  for  a 
definitive,  visible  contractor  SQA  program  and  associated 
planning  documents  £32:483.*  This  specification  addresses 
such  topics  as: 

1.  Tools,  techniques  and  methodologies.  Concerns 
whether  or  not  the  contractor  identified  and  de¬ 
fined  the  software  engineering  tools,  techniques 
and  methodologies  planned  to  support  the  require¬ 
ments  of  SQA. 

2.  Computer  program  design.  Concerns  how  the  SQA 
program  will  handle  procedures  for  the  review  and 
evaluation  of  software  design  documentation. 
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3.  Work  certification.  Concerns  the  contractor's 
procedures  for  formally  approving  the  completion 
of  work  performed  under  the  contract. 

4.  Documentation.  Concerns  how  the  SQA  program 
shall  insure  compliance  with  accepted  standards, 
practices  and  conventions  of  documentation  during 
software  development. 

5.  Testing.  Concerns  the  process  of  explicitly 
assuring  that  the  software  performs  as  required. 

Consequently,  MIL-S-52779A,  "...  has  generated  an  in¬ 
creased  awareness  of  the  need  for  SQA  in  software  develop¬ 
ment  [48:13." 

MIL-S-52779A  is  a  brief  document  the  contents  of  which 
are  further  explained  in  Military  Handbook  (MIL-HDBK)  334. 

MI L-HDBK-334  was  published  to  clarify  MIL-S-52779A  and  to, 

".  .  .  provide  guidance  for  the  evaluation  of  a  contractors 
SQA  program  when  MIL-S-52779A  is  invoked  C32:iv3."  This 
handbook  emphasizes  the  planning  and  execution  of  a  com¬ 
prehensive  SQA  program.  In  addition,  the  handbook  specifies 
that  a  contractor's  SQA  program  should  be  developed  to  iden¬ 
tify  deficiencies  early  in  the  development  process.  MIL- 
HDBK-334  also  notes  the  strong  relationship  between  configu¬ 
ration  management  and  SQA.  This  handbook  attempts  to 
clarify  issues  created  by  MIL-S-52779A. 

However,  while  the  DoD  has  attempted  to  improve  soft¬ 
ware  quality  through  software  quality  management  practices 
as  contained  in  the  above  directives,  regulations,  stand¬ 
ards,  and  specifications,  low  quality  software  continues  to 
be  developed.  Low  quality  software  is  characterized  by 
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“cost  and  schedule  overruns,  poor  performance  of  systems 
when  delivered,  high  maintenance  costs,  lack  of  reliability, 
and  a  high  degree  of  system  sensitivity  to  changes  in 
requirements  [29:13].“  Consequently,  it  is  argued  that 
while  DoD  documents  allow  for  the  existence  of  an  SQA  pro¬ 
gram,  an  SQA  program  in  and  of  itself  is  not  enough  to 
insure  the  development  of  quality  software. 

Summary 

According  to  the  1 i terature  reviewed,  timing  for  base¬ 
line  control  should  be  established  in  relation  to  engineer¬ 
ing  design  reviews  as  follows.  A  functional  baseline  should 
be  established  as  a  result  of  a  Systems  Requirements  Review 
(SRR).  The  functional  baseline  forms  the  basis  of  the 
System  Design  Review  (SDR).  After  the  conclusion  of  the 
SDR,  allocated  baselines  should  be  defined  and  established. 

A  PDR  follows  the  establishment  of  the  allocated  baseline 
which  then  serves  as  the  foundation  for  the  detailed  design 
work.  Draft  product  baseline  documentation  should  be  com¬ 
pleted  ahead  of  a  CDR.  After  CDR,  the  detail  design  is 
tested  and  the  draft  product  baseline  documentation  is  re¬ 
fined  and  completed.  Formal  establishment  of  the  product 
baseline  should  be  rendered  after  Physical  Configuration 
Audit  (PCA)  has  been  conducted. 

Uithin  the  scheme  of  this  management  plan,  a  strong 


commitment  to  establishing  these  baselines  at  these  specific 
points  in  the  program  is  essential.  Further,  the  program 
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manager  must  be  unwilling  to  progress  -from  one  baseline  to 
the  next  until  necessary  documentation  o-f  the  proper  -form 
and  content  is  available.  This  last  qualification  implies  a 
notion  o-f  quality  being  designed  into  the  product  as  it 
matures  -from  beginning  to  end. 

As  discussed  in  chapter  two,  the  concept  o-f  quality  as 
tradi  t  i  onal  1  y  applied  to  the  development  o-f  a  hardware  pro¬ 
duct,  "...  easily  lends  itself  to  quantifiable  standards 
£22:2173.*  However,  with  respect  to  the  development  of 
software,  “The  science  of  managing  software  development  is 
still  in  its  infancy,  and  the  lack  of  a  good  clear  set  of 
principles  is  apparent  £44:2513.*  Nevertheless,  in  re¬ 
searching  the  topic  of  quality,  as  related  to  the  develop¬ 
ment  of  a  software  product,  a  single  principle  did  emerge 
as  significantly  contributing  to  the  development  of  a  quality 
software  product.  Specifically,  the  idea  of  "designing-in* 
quality  throughout  the  course  of  the  Software  Development 
Life  Cycle  appeared  as  the  key  to  developing  quality  soft¬ 
ware.  Consequently,  the  end  result  supports  a  conclusion 
that,  for  the  development  of  a  quality  software  product, 
quality  must  be  designed  into  a  software  product  through  a 
formalized  set  of  controls-  controls  which  are  provided  by 
the  discipline  of  configuration  management. 

■Configuration  Management  provides  the  detailed  frame¬ 
work  upon  which  a  SQA  program  can  be  built  because  it  estab¬ 
lishes  the  formal  structure  necessary  to  enforce  compliance 
with  procedures  £44:73."  Spec i f i cal  1 y ,  through  a  series  of 
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engineering  reviews  and  con-figuration  audits  technical  and 
administrative  direction  are  applied  to  <a>  identify  and 
document  -functional  and  physical  characteristics  o-f  systems 
and  con-f iguration  items;  <b>  control  changes  to  those  charac 
ter i sties;  and  <c)  record  and  report  change  processing  and 
implementation  status  <20:6-1).  Consequently,  this  direc¬ 
tion  allows  -for  implementation  o-f  the  essential  SQA  support 
■factors —  o-f  securing  management  support  -for  the  development 
o-f  quality  so-ftware,  -formulating  an  independent  SQA  team  to 
ensure  the  development  o-f  quality  so-ftware,  conducting  su-f-f  i 
cient  testing  to  verify  software  quality,  and  ensuring  the 
next  phase  of  the  development  process  is  not  entered  into 
until  existing  phase  criteria  are  fulfilled.  Therefore,  in 
the  opinion  of  these  authors,  a  particularly  strong  relation¬ 
ship  exists  between  the  disciplines  of  SQA  and  baseline 
management.  A  relationship  which  is  not  utilized  to  its 
fullest  potential. 
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IV.  He  thodol ogy 

Introduct i on 

This  chapter  summarizes  the  methodology  used  in  the 
collection  and  analysis  o-f  data  necessary  to  answer  the 
thesis"  management  question.  Specifically,  the  concerns  to 
be  addressed  include  <1)  a  discussion  of  the  procedural 
design  of  the  research,  as  developed,  to  yield  the  most 
objective  results;  <2 )  a  discussion  of  reporting  procedures, 
with  emphasis  on  the  objectivity  of  reporting;  and  (3)  vali¬ 
dation  of  the  survey  instrument.  Further,  this  chapter  dis¬ 
cusses  what  was  expected  to  be  found;  the  statistical  tests 
chosen  to  test  data;  and  the  assumptions  made  in  formulating 
the  overall  design  of  this  research  effort.  This  chapter 
provides  a  detailed  outline  of  the  research  procedures  to 
allow  a  fair  replication  of  this  research. 

This  thesis'  primary  assertion  is  that  a  software  qual i 
ty  assurance  <SQA)  plan  together  with  baseline  management 
can  be  integrated  into  a  single  management  program  whereby  a 
software  acquisition  development  program's  cost  and  schedule 
integrity  can  be  effectively  maintained  and  the  computer 
program's  reliability  enhanced,  thereby  reducing  its  main¬ 
tenance  requirements  during  the  operational  phase  of  its 
life.  This  premise,  however,  presumes  that  both  SQA  and 
baseline  management  have  some  level  of  influence  over  the 
management  control  of  the  product.  Therefore,  in  explora- 
t i on  of  this  premise,  it  is  first  necessary  to  determine  if 
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SQA  and  baseline  management  independently  contribute  to 
program  integrity.  If  SQA  and  baseline  management  do  inde¬ 
pendently  influence  program  integrity,  this  in-formation  can 
then  be  used  to  probe  the  possible  joint  e-f-fect  o-f  SQA  and 
baseline  management  upon  program  integrity. 

Developing  the  Data  Gathering  Instrument 

The  design  o-f  a  longitudinal  experiment  allows  -for  an 
investigator  to  observe  the  reaction  o-f  a  test  group  to 
varying  stimuli  in  comparison  to  an  unexposed  control  group. 
In  consideration  o-f  the  time  allowed  to  conduct  this  re¬ 
search  and  the  nature  o-f  the  situation  to  be  explored,  it 
was  necessary  to  -find  an  alternative  means  to  collect  data 
other  than  a  -form  o-f  a  longitudinal  study.  Consequently, 
operating  within  a  limited  time  dimension  and  with  no  con¬ 
trol  over  the  variables  involved,  it  was  decided  to  develop 
this  research  e-f-fort  as  a  cross-sectional  field  study  within 
the  context  of  an  ex  post  facto  design.  Further,  under¬ 
standing  that  this  research  may  be  classified  as  primarily 
causal  in  nature,  it  was  necessary  to  attempt  to  uncover 
comparative  groups  which  have  and  have  not  been  exposed  to 
the  "causal*  factors  under  study. 

The  chosen  course  was  to  obtain  and  evaluate  historical 
data  through  the  use  of  a  questionnaire  as  administered 
through  the  process  of  a  structured  interview.  Using  the 
hypotheses  as  a  guide  for  the  direction  of  this  study,  the 
questionnaire  was  designed  and  constructed  to  obtain  needed 
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data.  The  design,  construction  and  use  of  the  questionnaire 
was  based  on  several  consi derat i ons. 

First,  the  criticisms  attributed  to  survey  method¬ 
ologies  were  reviewed.  Spec  i  f  i  call  y ,  while  complete  depen¬ 
dence  o-f  verbal  responses  may  result  in  untrue  or  misleading 
answers,  the  versatility  associated  with  an  interrogation 
process  outweighed  any  shortcomings  o-f  t.  i  s  method.  The 
personal  interview  was  viewed  as  the  most  practical  and 
economical  way  to  uncover  needed  information. 

Second,  a  one-on-one  interviewer-interviewee  relation¬ 
ship  was  pursued  to  take  advantage  o-f  the  greater  depth  and 
detail  of  information  that  could  be  secured,  as  well  as  the 
possibility  of  gathering  higher  quality  information  through 
noting  the  conditions  of  the  interview,  additional  probing, 
or  general  observation. 

Third,  the  questionnaire  was  developed  in  a  standard¬ 
ized  format  and  sequence  to  help  assure  that  each  question 
was  asked  in  the  same  way,  thus  promoting  measurement  relia¬ 
bility.  As  a  corol 1  ary  to  the  question  structure,  the 
response  structure  afforded  the  interviewee  was  also  con¬ 
sidered.  Consequently,  while  many  questions  could  be  an¬ 
swered  yes  or  no,  because  of  the  complexity  of  the  subject 
matter  respondents  were  given  the  opportunity  to  elaborate 
and  provide  qualifications  to  their  answers.  Therefore,  a 
more  thorough  understanding  of  any  explanation  could  be 
gained  while  minimizing  risk  of  any  misunderstanding. 

Lastly,  objectives  of  this  research  effort  were  not 
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disguised,  since  respondents  were  knowledgeable  at  a  con¬ 
scious  level  o-f  needed  in-formation  and  willing  to  provide 
i  n-f  or  mat  i  on  .  In  addition  to  these  considerations,  it  was 
also  -felt  that  a  questionnaire  used  in  conjunction  with  a 
structured  interview  would  provide  increased  continuity  in 
the  data  gathered  among  the  di-f-ferent  programs  investigated. 

Sample  Population 

This  research  e-f-fort  was  con-fined  to  programs  managed 
at  Aeronautical  Systems  Division  (ASD)  in  which  system  or 
subsystem  so-ftware  was  being  or  had  been  developed.  The 
scope  o-f  this  research  was  limited  to  system  so-ftware  -for 
the  purposes  o-f  standardization,  and  only  ASD  programs  were 
selected  to  prevent  possible  deviations  in  the  data  collected 
due  to  local  policies  and  practices  potentially  inherent  in 
other  Air  Force  System  Command  product  divisions.  Initial 
introduction  to  the  population  o-f  ASD  acquisition  programs 
was  obtained  through  a  computerized  list  maintained  by 
AFALC/XR,  which  identified  Deputy  Program  Managers  for  Logis 
tics  <DPMLs)  and  Integrated  Logistic  Support  Managers 
<ILSMs>  supporting  93  ongoing  program  efforts.  The 
DPMLy'I LSM  listing  was  deemed  to  be  a  convenient  means  of 
identifying  program  managers.  DPMLs  or  ILSMs  were  contacted 
and  were  requested  to  identify  ASD  program  managers  cui — 
rently  managing  software  development  programs.  ASD  program 
managers  were  also  identified  during  the  actual  interviewing 
process  when  respondents  were  asked  to  identify  other  pro- 
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grains  which  may  meet  the  criteria  o-f  this  research  effort. 
For  programs  currently  developing  software,  this  research 
e-f-fort  dictated  that  the  program  be  in  the  later  stages  o-f 
Full  Scale  Development,  so  that  a  sufficient  cost  and  sched¬ 
ule  history  would  be  available  to  correlate  to  the  existing 
SQA  and  baseline  control  programs. 

Since  later  candidate  interviews  were  obtained  through 
suggestions  of  interviewed  program  managers,  a  concern  arose 
that  the  sample/s  randcmness  would  be  impaired.  However, 
according  to  Mr.  Jeff  Daneman ,  AFIT/LSQ,  who  approved  of  the 
sampling  methodology,  while  many  texts  consider  randomness 
to  mean  that  each  element  of  a  population  has  an  equal 
chance  of  selection,  the  concept  of  randomness  also  means 
that  there  is  no  discrimination  in  selection  <11>.  There¬ 
fore,  because  every  referenced  program  was  contacted  and 
each  contacted  program  manager  who  was  willing  to  partici¬ 
pate  was  interviewed,  the  sampling  was  completed  without 
bias  and  can  be  considered  random. 

Planning  for  Control  of  Situational  Factors 

An  additional  concern  to  this  research  effort  was  the 
actual  interrogation  process  itself.  Viewed  as  critical  to 
the  success  of  this  research  were  Emory's  broad  criteria  for 
a  successful  personal  interview.  Emory's  criteria  are 
(24; 294) : 

l.  accessibility  by  the  respondents  to  needed 

i nf ormat i on , 
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2.  an  understanding  by  the  respondents  o-f  their 
roles,  and 

3.  motivation  o-f  the  respondents  to  accept  such 
a  role  and  to  fulfill  its  requ  i  remen  ts . 

Prescreening  o-f  the  respondents  prior  to  arranging  an  intei — 
view  appointment  insured  the  accessibility  o-f  respondents  to 
needed  information.  At  the  beginning  o-f  each  interview, 
every  effort  was  made  to  gain  a  good  interviewing  relation¬ 
ship.  Research  objectives  were  explained  in  an  attempt  to 
outline  the  role  of  the  respondent  and  to  motivate  the 
respondent  towards  an  understanding  of  the  importance  of 
this  study.  Additionally,  in  terms  of  actually  conducting 
the  interview,  all  interview  appointments  were  scheduled  at 
the  program  managers  convenience,  and  it  was  emphasized  that 
the  interview  was  to  take  a  limited  amount  of  time  and  that 
any  interruptions  would  be  understood.  During  the  inter¬ 
view,  each  question  was  read  directly  from  the  questionnaire 
and  then  clarified  with  respect  to  intent,  as  needed.  Re¬ 
sponses  were  recorded  during  the  interview  by  both  re¬ 
searchers  for  purposes  of  accuracy,  and,  after  each  inter¬ 
view  session,  recorded  responses  were  compared  to  identify 
and  clarify  any  recorded  deviations. 

Planning  for  Measurement 

Measurement  is  the  process  through  which  the  hypotheses 
in  this  thesis  will  be  tested.  However,  before  any  measure¬ 
ment  could  be  undertaken  with  the  questionnaire,  the  question¬ 
naire  had  to  be  validated.  Thorndike  and  Hagan  state  that 
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validity,  "...refers  to  the  extent  to  which  a  test  measures 
what  we  actually  wish  it  to  measure  (24:123;  46; 5).*  While 
many  methods  exist  to  validate  an  instrument,  the  validation 
of  the  survey  instrument  utilized  in  this  research  effort 
was  accomplished  through  expert  evaluation.  Spec  i -f  i  cal  1  y, 
both  the  data  gathering  instrument  and  data  gathering  ap¬ 
proach  were  discussed  with  experienced  individuals  within 
the  ASD  community  <4;  5;  7;  11>  respected  for  their  know¬ 
ledge  of  software  development  and  statistics. 

To  test  the  hypotheses,  the  collected  information, 
needed  to  include  a  description  of  a  program  in  terms  of  its 
(1)  SQA  program,  (2)  baseline  management  policy,  and  <3> 
cost  and  schedule  performance  track.  With  such  information, 
the  plan  was  to  substantiate  that  if  a  program  had  both  a 
sound  SQA  program  and  complied  with  the  baseline  management 
standard,  then  the  program  would  reflect  a  favorable  cost 
and  schedule  history.  Conversely,  a  program  which  did  not 
have  a  sound  SQA  program  or  employ  the  baseline  management 
standard  would  suffer  from  significant  cost  overruns  and 
schedule  slips.  While  concrete  examples  of  both  situations 
would  provide  the  cleanest  and  easiest  data  to  handle,  the 
reviewed  literature  suggested  that  no  such  clear-cut  situa¬ 
tion  exists.  Consequently,  to  ensure  that  all  programs 
would  be  treated  equally,  a  singular  method  to  handle  data 
was  established  prior  to  initiation  of  the  interviews  as  is 
subsequently  discussed. 

Software  Quality  Assurance  Program.  The  SQA  1 i terature 
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reviewed  in  chapter"*  two  and  three  concerned  both  non-DoD 
and  DoD  attempts  to  insure  the  development  of  quality  soft¬ 
ware.  Specifically,  chapter  two  emphasized  the  concepts 
that  serve  as  the  foundation  for  an  effective  SQA  program, 
while  chapter  three  reviewed  DoD  attempts,  through  various 
directives,  regulations,  military  standards,  and  military 
specifications,  to  operationalize  chapter  two  concepts 
within  the  DoD.  Consequently,  recalling  that  the  primary 
concern  associated  with  the  SQA  hypothesis  requires  a  deter¬ 
mination  of  the  soundness  of  an  SQA  program,  the  generic 
concept'  discussed  in  chapter  two  will  serve  as  the  primary 
source  for  this  determination  rather  than  the  specific  re¬ 
quirement  addressed  in  chapter  three. 

Programs  based  upon  the  concepts  presented  in  chapter 
two  were  considered  to  have  more  effective  (sound)  SQA  pro¬ 
grams  and  would  lead  to  the  eventual  production  of  quality 
software.  Specifically,  the  four  key  support  factors  discus 
sed  in  chapter  two  were  viewed  as  contributing  to  an  effect¬ 
ive  SQA  program.  The  first  support  factor  was  the  degree  of 
upper  level  management  support  by  the  contractor  producing 
the  software  and  by  the  program  office  procuring  the  soft¬ 
ware.  The  determination  of  such  support  was  made  based  on 
the  existence  or  non-existence  of  upper  level  management 
policies,  methodologies,  SQA  planning  documents,  standards, 
reviews  and  audits,  and  the  degree  of  commitment  to  such 
items.  The  second  key  support  factor  concerned  the  establish- 


50 


ment  o-f  an  independent  SQA  activity.  Within  a  program 
of  f  ice,  an  independent  SQA  activity  usually  takes  the  form 
of  an  Independent  Validation  and  Verification  team,  which  is 
a  group  of  either  Air  Force  engineers  or  privately 
contracted  software  engineers  attempting  to  verify  software 
performance.  Within  a  contractors  facility,  an  independent 
SQA  activity  was  considered  to  be  an  independent  function 
specifically  devoted  to  SQA.  The  third  key  support  factor 
concerned  testing.  The  testing  aspect  of  developing  quality 
software  dealt  with  how  the  testing  process  was  conducted, 
who  was  present  at  testing  and  how  discrepancies  were  docu¬ 
mented  and  tracked  through  to  completion.  The  final  support 
factor  explored  concerned  the  controlled  progression  of 
software  development,  where  the  position  taken  was  that  no 
new  phase  of  software  development  should  be  undertaken  until 
work  in  preceding  phases  was  accomplished  to  the  satisfac¬ 
tion  of  the  program  manager. 

Consequently,  from  these  support  factors,  which  were 
incorporated  in  the  questionnaire,  the  determination  of  the 
effectiveness  or  non-effectiveness  of  an  SQA  program  was 
made.  As  will  be  explained  in  chapter  five,  all  programs 
not  founded  in  the  above  mentioned  support  factors  were 
considered  to  have  an  unacceptable  (unsound)  SQA  program. 

The  term  unacceptable  only  means  that  the  program  SQA  plan 
did  not  appear  to  be  based  in  the  support  factors  espoused 
within  the  1 i terature  reviewed. 

Base  1 i ne  Control .  As  discussed  in  chapter  three,  AFR 
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800-14  was  the  primary  source  -from  which  the  appropriate 
timing  -for  establishment  of  the  respective  specification 
baselines  (functional,  allocated,  and  product)  was  deter¬ 
mined.  Spec i f ical 1 y,  if  a  program  had  an  established  func¬ 
tional  baseline  after  SRR  but  prior  to  SDR,  an  established 
allocated  baseline  pr i or  to  preliminary  design  review,  and 
an  established  product  baseline  with  the  completion  of  the 
physical  configuration  audit,  the  program  was  considered  to 
have  complied  with  the  baseline  management  standard.  All 
other  instances  were  considered  to  deviate  from  the  baseline 
management  standard. 

Significant  Cost  Overruns.  The  cost  hypothesis  sug¬ 
gests  that  a  significant  cost  overrun  will  occur  when  an 
unacceptable  SQA  or  baseline  management  policy  is  employed 
by  a  program.  Each  interviewee  was  asked  what  they  consider 
a  significant  cost  overrun  to  be  on  a  program,  as  a  percent¬ 
age  of  the  program  cost.  A  significant  cost  overrun  will 
then  be  determined  by  averaging  the  responses  of  interviewed 
program  managers. 

Actual  Program  Cost  Overruns.  The  actual  program  cost 
overrun  will  be  calculated  as  the  cost  of  engineering  change 
proposals  (ECP)  due  to  software  deficiencies  (excluding 
identified  deficiencies  because  of  new  requirements)  divided 
by  the  cost  of  software  in  the  program.  Thus,  a  J300K 
software  ECP  on  a  $1M  program  will  be  equivalent  to  a  *30K 
software  ECP  on  a  ♦100K  program;  each  equalling  30  percent. 
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When  the  mean  cost  overrun  is  being  calculated,  the  cost 
overrun  percentage  will  be  multiplied  by  a  basis  o-f  *100. 
For  example,  i  f  program  XYZ's  cost  overrun  percentage  is  15 
percent,  then  program  XYZ's  cost  overrun  will  be  calculated 
as,  *100  X  .15,  or  *15. 

Significant  Schedule  Slip.  In  each  interview,  the  pro¬ 
gram  manager  was  asked  to  specify  their  understand i ng  of  a 
significant  schedule  slip.  An  average  of  the  responses  will 
be  used  as  the  decision  rule  to  determine  if  a  program  does 
or  does  not  have  a  significant  schedule  slip.  Based  on  the 
preceding  decision  rules  the  cost  and  schedule  information 
can  be  classified  into  one  of  two  categories  for  each  of  the 
areas  being  studied. 

Statistical  Test 

For  the  type  of  information  expected  to  be  gathered,  an 
appropriate  statistical  measure  is  the  one-tailed  t  test  for 
small  sample  inferences  about  the  difference  between  two 
population  means.  A  small  sample  implies  a  sample  size  of 
less  than  30  observations  for  each  population  being  sampled. 
The  key  assumption  is  that,  "To  use  the  t  statistic,  both 
sampled  populations  must  be  approximately  normally  distri¬ 
buted  with  equal  variances,  and  the  random  samples  must  be 
selected  independently  of  each  other  136:3373.*  This 
assumption  is  made  based  on  the  understanding  that  the 
sampling  distribution  of  the  sample  means,  x  and  x  , 
depends  on  the  shape  of  the  populations  being  sampled.  The 
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use  o-f  the  t  statistic  involves  the  proposal  of  a  null 
hypothesis,  which  is  either  accepted  or  rejected  in  -favor  o-f 
an  alternative  hypothesis  based  on  a  selected  level  o-f 
si  gn  i -f  i  cance  .  The  t  statistic  is  o-f  the  -form 


= 

samp  1 e 

mean 

from 

samp  1 e 

1 

x  1 

=s 

samp  1 e 

mean 

form 

samp  1 e 

2 

n1 

= 

samp  1 e 

s  i  ze 

from 

samp  1 e 

1 

nj. 

ss 

samp  1 e 

size 

from 

samp  1 e 

2 

s? 

= 

pool ed 

sample  standard 

deviation 

<n-]  +  n2  -  2)  degrees  of  -freedom.  The  one-tailed  t  test 

would  be  used  to  determine  i -f  there  is  a  significant  differ¬ 
ence  in  the  mean  of  one  group  from  the  mean  of  the  other. 

The  t  test  is  the  test  method  which  will  be  used  to 
test  the  hypotheses  proposed  in  this  research  effort.  Pro¬ 
gram  cost  data,  obtained  from  program  managers,  was 
collected  for  each  program  and  a  mean  was  calculated.  This 
mean  value  was  than  incorporated  into  the  t  statistic  to 
test  the  following  hypotheses: 

Ho!  A  significant  cost  overrun  is 

independent  of  a  sound  SQA  program;  Ml  =  M2. 

Ha!  A  significant  cost  overrun  is  not  independent 
of  a  sound  SQA  Program;  Ml  <  M2. 

H  os  A  cost  overrun  is  independent  of  the 

baseline  management  standard;  Ml  =  M2. 

Ha!  A  cost  overrun  can  be  reduced  by  applying 
baseline  management  standard;  Ml  <  M2. 
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Likewise,  i  n  -forma  t  i  on  regarding  schedule  slips  were  obtained 
•for  each  program  and  a  mean  calculated  in  the  -following  t 
tests: 


Hq:  A  schedule  slip  will  be  independent 

o-f  a  sound  SQA  program;  M3  =  M4. 

H^:  schedule  slip  can  be  reduced  by  a 
a  sound  SQA  program;  M3  <  M4. 

Hc:  A  schedule  slip  will  be  independent 

o-f  the  baseline  management  standard; 
M3  =  M4. 

H^:  A  schedule  slip  can  be  reduced 

by  applying  the  baseline  management 
standard;  M3  <  M4. 


While  such  concepts,  as  implied  by  the  words  "sound”  and 
"standard",  are  di-f-ficult  to  quantify,  the  design,  construc¬ 
tion  and  validation  of  the  questionnaire  serves  as  the  basis 
by  which  such  concepts  are  defined.  As  explained  above, 
those  concepts  identified  in  the  literature  as  significantly 
contributing  to  an  effective  SQA  program  and/or  an  effective 
baselining  policy  were  incorporated  into  the  questionnaire. 
Based  on  such  criteria,  each  sample  program  was  evaluated  to 
determine  the  effectiveness  of  its  SQA  and  baseline  manage¬ 
ment  pol i c i es. 

The  final  aspect  of  this  research  methodology  concerns 
exploration  into  the  question: 

Can  a  SQA  program  and  a  basel ine  manage¬ 
ment  program  be  integrated  into  a  single  plan 
which  will  result  in  greater  benefits  than  the 
individual  application  of  either  of  these  methodo¬ 
logies? 

With  this  concern  being  primarily  exploratory  in  nature, 
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the  purpose  was  to  develop  concepts  more  clearly  and  pro¬ 
vide  greater  insights  into  the  relationships  between  the 
variables  of  SGA  and  baseline  managemers  t  .  The  methodology- 
used  was  an  opinion  survey  relating  to  the  need  -for  some 
relationship  to  exist  between  SQA  and  baseline  management. 
The  individual  responses  will  be  resorted  in  chaster  -five. 

Summary 

This  chapter  has  described  the  research  me thodol ogy 
used  -for  this  research  e-f-fort.  The  intent  was  to  describe 
the  thesis  methodology  in  sufficient  detail  so  as  to  permit 
a  reader  to  accomplish  a  complete  replication  o-f  this 
research.  This  chapter  included  a  discussion  o-f  the  de¬ 
sign,  construe  t  i  on  ,  validation  and  utilization  o-f  the 
questionnaire  within  the  context  o-f  a  structured  interview 
format  and  a  discussion  of  the  statistical  test,  associated 
assumptions  and  the  sampling  technique  utilized. 

This  effort  was  proposed  as  a  cross-sectional  study  o-f 
an  ex  post  facto  design.  The  primary  assertion  of  this 
research  effort  is  that  an  SQA  plan  together  with  baseline 
management  can  be  integrated  into  a  program  whereby  a  soft¬ 
ware  program's  cost  and  schedule  integrity  can  be  main¬ 
tained,  the  computer  program's  reliability  enhanced,  and 
its  maintenance  requirements  reduced  during  the  operational 
phase  of  its  life.  In  the  next  chapter,  data  handling  and 
analysis  are  discussed. 


V.  Findings  and  Analysis 


Introduct i on 

Analysis  of  the  collected  data  is  presented  in  this 
chapter.  This  analysis  will  be  presented  in  two  stages. 
First,  a  discussion  o-f  collected  data  will  be  conducted  to 
-familiarize  the  reader  with  the  actual  survey  instrument 
utilized;  describing  the  rationale  used  to  decide  if  a  par — 
ticular  program  should  be  classified  as  having  a  sound  or 
an  unsound  SGA  program  and  as  following  or  not  following 
the  baseline  management  standard,  and  summarizing  classifi¬ 
cation  findings.  Second,  having  categorized  the  findings, 
the  test  statistic  will  be  presented  and  a  determination 
made  to  either  accept  or  reject  the  hypotheses.  Conclu¬ 
sions  and  recommendations  derived  from  this  analysis  will 
be  presented  in  the  subsequent  chapter. 

Data  Collection 

The  collected  data  consists  of  twenty-one  interviews 
with  Program  Managers  <PMs>.  All  interviewed  PMs  were  man¬ 
aging  programs  requiring  the  development  of  software  for 
Embedded  Computer  Systems  with  the  program  having  at  least 
progressed  well  into  Full  Scale  Development.  The  survey 
instrument  utilized  throughout  the  course  of  the  interview¬ 
ing  process  is  presented  in  Appendix  A.  The  questions 
contained  within  the  instrument  were  designed  to  explore 
the  key  aspects  of  a  program  wi th  respect  to  SQA  and  base¬ 
line  management.  Each  interview  is  classified  in  terms  of 
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TABLE  VI 1 


Program  Classification  Summary 


Sound 

Standard 

Program 

SQA 

Base  1 i ne 

Comments 

1  . 

No 

No 

2. 

No  answer  No 

3. 

No  answer  No 

4. 

Yes 

No 

Not  used; 

see  program 

anal ysi s 

in  Appendix  B 

5. 

No 

No 

6. 

No 

No 

7. 

Yes 

No 

8. 

No 

Yes 

9. 

No 

No 

Not  used; 

see  program 

anal ysi s 

in  Appendix  B 

10. 

N/A 

N/A 

Not  used; 

see  program 

anal ysi s 

in  Appendix  B 

11  . 

No 

No 

12. 

Yes 

No 

13. 

No 

No 

14. 

Yes 

No 

15. 

Yes 

Yes 

16. 

No 

Yes 

17. 

No 

Yes 

18. 

No 

No 

19. 

No 

Yes 

20  . 

No 

Yes 

21  . 

No 

No 

both  SQA  and  baseline  management.  Specifically,  a  program 
is  classified  as  either  having  a  sound  or  unsound  SQA 
program  and  as  following  or  not  following  the  baseline 
management  standard.  The  rationale  for  these  classifica¬ 
tions  is  contained  in  Appendix  B.  In  addition.  Appendix  B 
also  contains  cost  and  schedule  information,  which  is  used 
for  input  into  the  test  statistic.  Table  VII  summarizes 
the  SQA  and  baseline  management  classifications  as  con¬ 
tained  in  Appendix  B. 
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TABLE  VIII 


Significant  Cost  Overrun  and 


Program  Managers'  Opinions  of  What  Constitutes 
Cost  Overruns  and  Schedule  Slips 


Cost  Cost 


Program 

Over¬ 

run 

Schedul e 

SI  ip 

Program 

Ovei — 
run 

Schedul e 
SI  ip 

1  . 

15 V. 

3  mo 

11  . 

NR 

NR 

2. 

NR 

NR 

12. 

30% 

2  wk 

3. 

NR 

NR 

13. 

20% 

NQ 

4. 

5/. 

NQ 

14. 

15% 

3  mo 

5. 

10% 

NQ 

15. 

NR 

NR 

6. 

25% 

3  mo 

16. 

10% 

NQ 

7. 

20% 

NQ 

17. 

15% 

4  mo 

8. 

NQ 

NQ 

18. 

10% 

5  mo 

9. 

15% 

3  mo 

19. 

15% 

4  mo 

10. 

2% 

3  mo 

20. 

10% 

NQ 

::  No  response. 

!:  Response  was 

not  quantifi 

iable.  See 

Appendi x 

B  for 

expl anat i on . 


Only  nine  of  the  21  interviews  resulted  in  useable 
responses  to  find  an  average  for  significant  a  schedule 
slip.  Based  on  the  nine  quantifiable  responses,  on  the 
average,  3.11  months  is  considered  a  significant  schedule 
slip.  It  should  be  noted  that  the  indicated  schedule  slips, 
contained  in  Table  VIII,  are  in  relation  to  end  item  de¬ 
livery  dates.  Also,  many  of  the  program  managers  caveated 
there  estimate,  saying  that  a  schedule  slip  is  really  only 
crucial  or  significant  when  the  user  or  mission  is  impacted. 
Therefore,  for  schedule  slips,  a  slip  beyond  the  needed 
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delivery  date  was  considered  a  better  measure  than  a  sched¬ 


ule  slip  as  a  percentage  of  the  contract  schedule  period. 

For  example,  a  three  month  slip  on  a  12  month  program  may 
not  have  the  same  impact  that  a  one  month  slip  would  have  on 
a  compressed  36  month  program  with  an  urgent  user  require¬ 
ment.  Conversely,  a  one  month  slip  on  a  critical  12  month 
program  may  cause  serious  repercussions  while  a  three  month 
slip  on  a  less  crucial,  36  month  program  would  have  no 
impact  at  all.  Uhat  needs  to  be  accounted  -for  is  the  urgen¬ 
cy  o-f  the  program  and  the  user's  need  -for  the  end  item. 

Fifteen  of  the  21  program  managers  interviewed 
provided  useable  estimates  for  calculation  purposes  of  what 
they  considered  to  be  a  significant  cost  overrun.  The 
average  of  these  responses  is  14.46  percent.  Appendix  B 
provides  additional  insights  as  to  what  constitutes  a 
significant  cost  overrun  or  schedule  slip. 

Statistical  Analysis 

As  presented  in  the  chapter  four  of  this  thesis,  the 
test  statistic  to  be  utilized  in  the  determination  of 
either  accepting  or  rejecting  thesis  hypothesis  was  the 
one-tailed  t  test  for  small  sample  inferences  about  the 
difference  between  two  population  means.  This  statistic  is 
of  the  form: 
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TABLE  IX 


SQA  Cost  Data  Summary 


Sound  SQA  Unsound  SQA 


’rogram 

Overrun 

Program 

Overrun 

7  . 

*  0 

1  . 

*11 

12. 

0 

5. 

0 

14. 

13 

6. 

7.2 

15. 

0 

8. 

0 

11 . 

0 

13. 

0 

16. 

0 

17. 

14 

IS. 

0 

19. 

0 

20. 

0 

21  . 

0 

Total 

*13 

Total 

*32.2 

= 

3.25 

~x\  = 

2.683333 

si  = 

5.629 

si  = 

4.85143 

ni  ~ 

4 

= 

12 

SP  = 

24.135514 

5.  Decision.  Since  the  calculated  value  is  less 
than  the  critical  value  <  .1997838  < 

1.761),  the  null  hypothesis  is  accepted. 

Showing  that  a  sound  SQA  program  could  cause  a  reduc- 
t i on  in  significant  cost  overruns  was  the  initial  reason 
for  this  test.  After  the  data  was  collected,  however,  the 
mean  cost  overrun  of  the  programs  wi th  a  sound  SQA  program 
was  greater  than  the  mean  of  programs  not  having  a  sound 
SQA  program.  Thus,  immediately  a  judgement  can  be  made 
that  the  data  does  not  support  the  premise  that  a  sound  SQA 
program  will  reduce  cost  overruns.  However,  a  test  was 
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carried  out  to  determine  if  enough  evidence  was  collected 
which  would  indicate  that  a  sound  SQA  program  would 
actually  have  an  adverse  effect  on  program  costs.  As  the 
test  shows,  however,  there  is  no  statistical  evidence  that 
a  sound  SQA  program  has  an  adverse  impact  on  program  costs. 

SQA  Schedule  Analysis. 

Table  X 

SQA  Schedule  Data  Summary 


Sound  SQA 


Unsound  SQA 


Program  SI  i  o 

7.  0  mo 

12.  0 

14.  0 

15.  0 


Total  0  mo 


Pr  ogr  am 
1 . 

5. 

6. 

8. 

11  . 
13. 
16. 

17. 

18. 
IP. 
20. 

21  . 

Total 


SI  i  p 
0  mo 
0 
0 
0 
0 
7 
12 
12 
0 
6 
7 
0 

44  mo 


*1  “  0 
s1  *  0 
n  |  *=  4 


x  L  =  3.6666667 
sL  =  4.6607105 
n  L  =  12 

17.06746 


1.  Null  hypothesis.  H0:  A  schedule  slip  will  be 

independent  of  a  sound  SQA 
program;  M3  =  M4. 

H^s  A  schedule  slip  can  be 
reduced  by  a  sound  SQA 
program;  M3  <  M4. 

Reject  H0  if  t  <  critical  -t  value. 
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2.  Significance  level.  alpha  *  0.05  (one-tailed 
test) 

3.  Calculated  value. 


t 


0  -  3.6666667 


17.06744 


-3.6646647 

2.3851946 

=  -1.537261  with  d.f.  =  14 

4.  Critical  test  value.  Critical  t  value  =  -.1761 

5.  Decision.  Since  the  calculated  value  greater  than 
the  critical  value  <  -1.537261  >  -1.761  >,  the 
null  hypothesis  is  accepted. 

Based  on  the  test  result,  there  is  no  statistical 
evidence  that  a  sound  SQA  program  effects  better  schedule 
management . 

■  ne  Cost  Analysis. 

1.  Null  Hypothesis.  Hp  :  A  significant  cost  overrun 

is  independent  of  the 
baseline  management 
standard;  Ml  ■  M2. 


H ^ :  A  significant  cost  overrun 
will  occur  by  applying  the 
baseline  management 
standard;  Ml  >  M2. 


Reject  H q  i f  t  >  critical  t  value. 


2.  Significance  level 
test) 


alpha  *  0.05  (one-tailed 


3.  Calculated  value 


22.014556 


TABLE  XI 

Baseline  Cost  Data  Summary 


Base] i ne 

Not 

Basel i ne  j 

Standard 

Standard 

Program 

Overrun 

Program 

Overrun 

15. 

*  0 

1 . 

*11  j 

16. 

0 

2. 

0  1 

17. 

14 

3. 

0  i 

19. 

0 

5. 

0 

20. 

0 

6. 

7.2 

7. 

0 

j 

8. 

0  i 

11 . 

0  | 

12. 

0 

13. 

0 

14. 

13  ] 

18. 

o  ; 

21  . 

0  i 

I 

t 

Total 

*14 

Total 

*31.2 

1 

*i  * 

2.8 

xz  = 

] 

2.6  ! 

s  = 

5.6 

5t  — 

4.6612 

n1  - 

5 

"i  = 

13  j 

Sp  *  22. 

014556 

s 

.2 

2.4691 

m  .081  with  d.f.  =  16 

Critical  test  value.  Critical  t  value  =  1.746 

Decision.  Since  the  calculated  value  is  less  than 
the  critical  value  (  .081  <  1.746),  the 
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The  initial  intent  for  this  test  was  to  show  that  the 


baseline  management  standard  would  contribute  to  better 
management  o-f  program  costs.  However,  because  the  mean 
cost  overrun  o-f  those  programs  adhering  to  the  baseline 
management  standard  was  greater  than  the  mean  o-f  the  other 
programs,  this  could  not  be  tested  for.  Instead,  a  test  was 
completed  to  determine  if  the  data  would  provide  sufficient 
evidence  that  application  of  a  baseline  management  standard 
would  adversely  impact  program  cost.  However,  based  on  the 
statistical  test  result,  it  must  be  concluded  that  a  cost 
overrun  is  independent  of  the  baseline  management  policy 
f ol 1  owed. 

Baseline  Schedule  Analysis. 

1.  Null  Hypothesis.  H0:  A  significant  schedule  slip 

is  independent  of  applying  a 
baseline  management  standard; 
Ml  =  M2. 

H^:  A  significant  schedule  slip 
will  result  by  applying  the 
baseline  management  standard; 
Ml  >  M2. 

Reject  if  t  >  critical  t  value. 


2.  Significance  level.  alpha  *  0.05  <one-tailed 
test) 


3.  Calculated  value. 

t  * 


7.4  -  .5384415 


7.3494441 


44 


4.8415385 


1 .4406403 

*  4.76283  with  d.f.  *  16 

4.  Critical  test  value.  Critical  t  value  *  1.746 

5.  Decision.  Since  the  calculated  value  is  greater 

than  the  critical  value  <  4.76283  > 
1.746),  the  null  hypothesis  is  rejected. 


TABLE  XII 

Baseline  Schedule  Data  Summary 


Basel ine 
Standard 


Not  Basel ine 
Standard 


Prooram  SI  i  d 

Program 

SjJJL 

15.  0  mo 

1 . 

0  mo 

16.  12 

2. 

0 

17.  12 

3. 

0 

19.  6 

5. 

0 

20.  7 

6. 

0 

7. 

0 

8. 

0 

11 . 

0 

12. 

0 

13. 

7 

14. 

0 

18. 

0 

21  . 

0 

Total  37  mo 

Total 

7  mo 

-  7.4 

X 

i  ~ 

.5384615 

s,,  *  4.4542 

s 

j.  * 

1 .865285 

nl  *  5 

n 

2  = 

13 

7.3494661 

a— 

The  initial  intent  of  this  test  was  to  provide  suffi¬ 
cient  evidence  that  application  of  the  baseline  management 
standard  would  curtail  a  significant  schedule  slip. 

Because  the  mean  of  those  programs  applying  the  baseline 


management  standard  was  greater  than  the  mean  of  programs 
not  applying  the  baseline  management  standard  this 
hypothesis  was  immediately  abandoned.  Instead,  a  test  was 
conducted  to  determine  if  application  o-f  the  baseline 
management  standard  will  cause  schedule  slips.  Based  on 
the  statistical  test,  it  appears  that  adherence  to  the 
baseline  management  standard  does  have  an  adverse  impact  on 
program  schedules. 

Software  Quality  Assurance  and  Baseline  Management  Combined 
With  the  analysis  of  SQA  and  baseline  management  com¬ 
pleted,  the  effect  of  combining  these  disciplines  will  now 
be  discussed.  The  hypothesis  proposed  implied  that  the 
combined  effect  of  SQA  and  baseline  management 
significantly  impacted  cost  and  schedule  adherence  during 
the  software  development  process.  Specifically,  the 
hypothesis  stated  that,  through  a  combination  of  SQA  and 
baseline  management,  a  synergistic  effect  resulted  which 
would  further  ensure  a  program  would  remain  within  cost  and 
on  schedule.  Consequently,  this  benefit  was  explored  with 
the  data  gathering  instrument. 

While  it  was  possible  to  gather  cost  and  schedule 
information  on  programs,  studying  the  combined  relationship 
between  SQA  and  baseline  management  proved  more  difficult. 
Consequently,  this  facet  of  the  research  may  be  classified 
as  primarily  exploratory  in  nature,  with  an  objective  of 
seeking  program  manager  ideas  on  the  important  issues  and 


aspects  of  this  subject  area 


Through  the  use  o-f  the  data  gathering  instrument, 
in-formation  was  obtained  -from  program  managers  on  the 
potential  synergistic  e-f-fect  o-f  combining  SQA  and  baseline 
management.  The  inputs  indicated  overwhelmingly  that  a 
definite  need  exists  for  some  form  of  a  relationship 
between  the  groups  responsible  for  SQA  and  baseline 
management.  However,  while  a  definite  need  was  expressed 
for  same  type  of  interaction  between  SQA  and  baseline 
management,  not  a  single  PM  could  identify  any  specific 
activities  or  constraints  which  could  inspire  that 
synergistic  effect  and  further  insure  that  a  software 
product  would  be  developed  within  cost  and  on  schedule.  A 
need  exists  for  type  of  interaction  between  SQA  and 
baseline  management,  but  further  research  is  required  to 
precisely  define  that  interaction. 

Summary 

Having  to  reject  an  hypothesis  is  disappointing,  but 
it  i s  as  important  to  show  that  there  exists  no  substantial 
evidence  to  support  commonly  held  beliefs.  This  does  not 
mean  that  such  assertions  are  incorrect,  only  that  there  is 
no  data  within  this  sample  which  provides  statistically 
valid  support.  Originally,  the  research  was  designed  to 
show  that  sound  SQA  and  application  of  the  baseline  manage¬ 
ment  standard  would  each  contribute  independently  to 
minimizing  cost  overruns  and  reducing  schedule  slips.  At  a 
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95  percent  confidence  level,  only  -for  the  e-f-fect  o-f  SQA  on 
schedules,  could  a  statistical  test  o-f  this  expected  posi¬ 
tive  relationship  be  structured.  Even  then,  there  is  no 
statistically  signi-ficant  support  in  -favor  o-f  this  positive 
rel at i onsh i p . 

For  the  other  three  statistical  test  groups,  which 
were  considered  to  possess  the  -favorable  characteristic,  it 
was  immediately  shown  that  no  such  positive  relationship 
can  be  expected.  In  -fact,  the  opposite  e-f-fect  is 
suggested.  There-fore,  testing  -for  each  o-f  these,  also  at 
the  95  percent  con-fidence  level,  was  completed  to  determine 
i-f  SQA  has  an  adverse  impact  on  cost  and  if  baseline 
management  has  an  adverse  impact  on  cost  and  schedule 
integrity.  The  test  results  indicate  that  only  one  of 
these,  the  adherence  to  the  baseline  management  standard  on 
schedule,  has  a  negative  effect.  However,  this  finding  is 
questionable.  As  is  explained  in  the  next  chapter,  these 
schedule  slips  may  be  explained  by  other  variables. 
Therefore,  the  general  conclusion,  based  on  the  statistical 
analyses  at  a  95  percent  confidence  level,  is  that  cost 
overruns  and  schedule  slips  will  occur  independently  of  a 
sound  SQA  program  or  adherence  to  the  baseline  management 
standard. 

In  summary,  data  analysis  in  this  chapter  found  that 
on  the  average  14.46  percent  over  target  price  is 
considered  a  significant  cost  overrun.  (An  interesting 
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statement,  made  by  two  program  managers,  is  that  there  is 
no  such  thing  as  a  cost  overrun,  only  contract 


modifications.)  On  the  average,  3  months  beyond  target 
delivery  date  is  considered  a  significant  schedule  slip. 
However,  almost  every  response  was  caveated,  saying  that  a 
slip  was  really  significant  only  when  the  mission  was 
impacted.  One  program  manager  commented  that,  while  no 
apparent  mission  impact  nu.y  be  realized  when  a  slip  occurs, 
there  is  the  undesirable  cost  of  maintaining  the  old 
equ i pment . 

In  this  chapter,  it  was  also  shown  that,  in  general, 
SQA  and  baseline  management  are  believed  to  share  a  close 
relationship.  No  empirical  data  could  be  obtained  to  test 
for  the  effect  of  such  a  relationship.  Finally,  except  for 
the  suggested  adverse  effect  of  the  baseline  management 
standard  on  schedule,  the  statistical  analyses  done  in  this 
chapter  indicates  that  cost  overruns  and  schedule  slips 
occur  independent  of  SQA  and  baseline  management.  However, 
while  the  tested  data  failed  to  substantiate  positive 
effects  of  SQA  and  baseline  management  on  software  cost  and 
schedule  control,  this  report  would  be  incomplete  without 
evaluating  other  available  data  to  assess  if  other  inter¬ 
vening  variables  might  have  biased  the  findings.  The 
discussion  of  such  analyses  and  findings  are  the  next 
chapter's  subjects. 
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VI .  Conclusions  and  Recommendations 

Evaluation  of  Research  Accomplishment 

The  exorbitant  cost  o-f  software,  reported  by  high  level 
Department  o-f  Defense  (DoD)  officials,  inspired  this  re¬ 
search  effort.  These  critics  attribute  the  high  software 
acquisition  and  maintenance  costs  to  poor  management  in  the 
dvv« I opment  phase.  The  results  are  development  cost  over¬ 
runs  and  delivery  of  poor  quality  software  that  is  ineffect¬ 
ive  and  difficult  to  maintain.  The  challenge  put  forth  to 
procure  affordable,  reliable,  and  maintainable  software 
provided  the  focus  and  direction  for  this  project. 

A  review  of  topical  1 i terature  emphasized  the  contribu¬ 
tions  of  SQA  and  configuration  management  to  effective  con¬ 
trol  of  software  development  and  the  extended  benefits  into 
the  operations  and  maintenance  phase.  This  1 i terature  also 
provided  a  framework  for  understanding  what  constitutes  a 
sound  SQA  program  and  the  development  of  a  standard  baseline 
management  scheme.  Comprehension  of  these  two  factors  was 
important  in  developing  the  test  hypotheses  and  the  data 
gathering  instrument. 

As  discussed  in  chapter  one,  the  time  constraint  re¬ 
stricted  the  study  to  the  effects  of  a  Software  Quality 
Assurance  (SQA)  program  and  the  application  of  a  standard¬ 
ized  baseline  management  plan  only  during  the  development 
phase.  Investigation  for  the  effects  of  these  two  variables 
on  the  operations  and  maintenance  phase  are  left  for  follow- 
on  work.  The  objective  of  this  thesis  was  to  obtain  the 
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evidence  necessary  to  substantiate  that  SQA  and  baseline 
management  independently  contribute  to  effective  management 
and  that,  together,  they  result  in  even  greater  control  over 
cost  and  schedule  integrity. 

The  data  was  collected  and  analyzed  in  an  unbiased 
manner.  Deviations  were  highlighted  by  the  researchers  and 
any  exceptions  taken  were  appropr i atel y  explained.  The 
individual  test  results  were  derived  -from  small  samples. 

Any  generalizations  made  are  only  representative  of  programs 
managed  at  Aeronautical  System  Division  and  can  only  be 
related  to  the  development  phase.  These  results  should  not 
be  used  to  presume  any  conclusions  about  circumstances 
involving  cost  or  schedule  in  the  operations  and  maintenance 
phase . 

The  researchers  were  able  to  complete  their  entire 
project.  However,  one  shortcoming  o-f  this  effort  is  that  no 
empirical  data  could  be  collected  to  evaluate  the  impact  o-f 
a  combined  SQA  and  baseline  management  program.  Although 
responses  -from  each  interview  indicated  with  universal 
agreement  that  such  a  combination  would  have  a  positive, 
synergistic  e-f-fect,  and,  indeed  that  it  does  or  should 
exist,  no  evidence  o-f  such  a  relationship  being  present  in 
any  o-f  the  programs  was  obtained.  While  none  o-f  the  test 
results  supported  any  o-f  the  expected  relationships  and  even 
suggests  that  a  negative  schedule  impact  can  result  -from 
-following  existing  guidance,  this  information  is  useful  for 
planning  purposes. 
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.  .  .  Serendipity —  the  exciting  occurrence 
of  -finding  useful  or  agreeable  ideas  or 
experiences  that  were  not  being  sought.  Often, 
the  important  advances  in  science  and  other 
disciplines  cane  unexpectedly.  An  experiment 
fails,  or  an  unusual  and  unexpected  result  is 
obtained,  and  to  the  alert  researcher  this  may 
start  a  new  train  of  thought  that  leads  to  a  new 
and  important  discovery.  The  Key  here  is  that  the 
researcher  must  retain  at  all  times  a  sensitive 
curiosity  which  is  stimulated  to  examine  these 
unexpected  results.  Moreover,  his  interest  and 
perspective  must  be  broad  enough  to  visualize  the 
possible  social  or  scientific  significance  of  the 
unusual  fact  that  the  discovery  may  portend 
C  33 : 473 . 

Software  Quality  Assurance.  As  evaluated  through  the 
course  of  this  research,  SQA  for  a  particular  program  has 
been  reviewed  with  regard  to  impact  on  program  cost  and 
schedule  integrity.  Consequen tl y ,  from  the  statistical 
analysis  presented  in  chapter  five,  a  determination  was  made 
that,  regardless  of  the  excellence  of  SQA  for  a  given  pro¬ 
gram,  the  established  cost  and  schedule  for  a  program  remain 
unaffected.  However,  while  the  quality  of  an  SQA  program 
did  not  appear  to  affect  program  cost  and  schedule,  several 
key  observations  may  be  made  concerning  the  degree  of 
effectiveness  of  an  SQA  program. 

Observations;  Sound  SQA  Programs. 

1.  In  terms  of  management  support,  the  fact  that  the 


Program  Manager  <PW>  had  upper  level  AF  management  support 
did  not  appear  significant.  However,  upper  level  management 
support  within  a  contractor's  organization  did  appear  to 
reduce  problems  associated  with  software  development  and 
assist  in  producing  a  higher  quality  software  product. 


2.  The  amount  and  variety  of  experience  o-f  the  PM,  in 
terms  o-f  years  o-f  program  management  and  software  programs 
previously  managed,  appeared  to  influence  the  overall  quali¬ 
ty  o-f  the  SQA  program.  Experience  assisted  the  PM  in  the 
recognition  o-f  key  facets  of  software  development  which 
contribute  to  both  the  development  of  a  quality  software 
product  and  the  on-schedule  and  within-cost  production  of  a 
software  product. 

3.  The  SQA  programs  classified  as  “Sound"  all  posses¬ 
sed  what  PMs  described  as  “workable"  SQA  plans  from  which 
the  intent  of  an  SQA  program  could  be  developed.  Further, 
for  “Sound"  SQA  programs,  PMs  had  usually  investigated  a 
contractor's  SQA  program,  personally  reviewed  the  contract¬ 
or's  proposed  SQA  plan  and  then  worked  together  with  the 
contractor  to  develop  what  a  PM  would  consider  a  “realistic” 
SQA  plan. 

4.  When  program  requirements  and  funding  permitted  the 
AF  to  contract  an  IV&V  team,  the  contributions  of  this  team 
generally  contributed  significantly  to  the  development  of  a 
higher  quality  software  product  and  to  a  software  product 
being  produced  both  on  schedule  and  within  cost. 

5.  The  contractor's  software  quality  assurance  section 
also  made  significant  contributions.  An  independent  SQA 
section  within  a  contractor's  organization,  especially  when 
supported  by  upper  level  management,  appeared  a  key  factor 
in  positively  contributing  to  software  development  efforts 
through  the  early  detection  and  correction  of  errors. 
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6.  For  software  development  efforts  classified  as 
•Sound* ,  a  contractor  policy  focusing  on  the  early  detection 
and  correction  of  software  deficiencies  appeared  signifi¬ 
cant.  The  primary  reason  given  by  PMs  for  contractor  imple¬ 
mentation  of  such  a  policy  was  that  contractor  upper  level 
management  appeared  to  recognize  potential  cost  savings 
derived  from  the  early  detection  and  correction  of  software 
di screpanc i es. 

7.  For  all  PMs  interviewed,  all  formal  and  informal 
reviews  and  audits  were  generally  conducted  in  a  satisfacto¬ 
ry  manner.  Further,  in  several  instances  where  a  contractor 
was  not  prepared  for  a  review  and  the  review  was  not  accomp¬ 
lished  to  the  satisfaction  of  the  PM,  a  PM  would  not  permit 
software  development  progression  until  a  rescheduled  review 
was  sat i sf ac tor i 1 y  accomplished. 

Observations:  Unsound  SQA  Programs. 

1.  Programs  in  the  "Unsound"  category  generally  had 
PMs  who  were  inexperienced  with  software  development  ef¬ 
forts.  Consequently,  a  recognition  of  the  potential  of  SQA 
did  not  exist,  and  a  lack  of  emphasis  concerning  SQA  was 
evident.  Further,  such  PMs  were  often  unaware  of  even  the 
most  general  contents  of  their  program/s  SQA  plan,  to  the 
extent  that  some  PMs  were  not  even  aware  of  the  existence  of 
an  SQA  plan  for  their  program. 

2.  SQA  plans  for  "Unsound"  programs  were  generally 
viewed  by  PMs  as  "show"  items  and  not  supported  by 
contractor  upper  level  management.  In  some  cases,  an  SQA 
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plan  did  not  exist  at  all.  For  existing  SQA  plans  in  this 
category,  PMs  often  stated  that  the  SQA  plan  on  their  pro¬ 
gram  was  only  given  a  cursory  review. 


3.  On  several  programs,  the  interaction  between  the 
contractor's  SQA  activity  and  the  contractor's  engineering 
activity  was  of  concern.  Specifically,  some  contractor's 
had  newly-formed  SQA  functions  which  handled  responsibili¬ 
ties  previously  handled  by  the  engineering  function.  Some 
hostility  existed  between  these  two  sections  which  resulted 
in  a  lack  of  cooperation  and  detracted  from  the  overall 
software  development  effort. 

4.  For  “Unsound*  SQA  programs,  the  contractor's  atti¬ 
tude  was  generally  one  that  did  not  emphasize  the  early 
detection  of  software  def i c i enc i es.  Reasons  provided  by  PMs 
for  this  attitude  included  existing  budget  and  schedule 
pressures  and  a  lack  of  upper  level  management  concern  by 
the  contractor. 

Observations;  General. 

1.  The  most  significant  general  observation  noted  b> 
these  researchers  concerned  the  definition  of  Software  Qual¬ 
ity  Assurance  as  interpreted  by  FM's.  With  regard  to  hard¬ 
ware,  the  1 i terature  reviewed  was  precise  in  defining  quali¬ 
ty  assurance  and  how  it  is  derived.  However,  when  it  comes 
to  software,  confusion  appears  to  exist  at  the  PM  level  as 
to  what  is  SQA  and  how  it  is  obtained  or  accomplished. 

2.  The  degree  of  software  development  experience  pos¬ 
sessed  by  a  PM  appeared  to  influence  the  quality  of  a  soft- 
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ware  product  and  program  cost  and  schedule  considerations. 
Specifically,  with  increased  experience  comes  an  increase  in 
emphasis  on  SQA  and  an  increase  in  recognition  of  those 
facets  which  contribute  to  the  production  of  a  quality 
software  product. 

3.  According  to  the  literature,  a  reader  concerned 
with  the  topic  of  AF  SQA  is  left  with  the  impression  that 
MIL-S-52779A  is  a  significant  part  of  the  answer  in  solving 
software  development  problems  in  the  AF.  However,  even  in 
those  SQA  programs  considered  "Sound",  MIL-S-52779A  was  not 
always  used.  Nor  did  the  SQA  plans  for  "Sound"  software 
development  programs  always  contain  all  the  elements 
presented  for  review  in  MIL-S-52779A. 

4.  As  presented  in  the  literature,  Software  Metrics  is 
considered  by  the  scientific  community  as  in  the  beginning 
stages  of  development.  This  conclusion  was  evident  through¬ 
out  the  course  of  all  the  interviews  conducted.  Specific¬ 
ally,  on  only  three  of  the  19  programs  reviewed  were  Soft¬ 
ware  Metrics  used  and  then,  according  to  the  respective  PMs, 
Metrics  were  used  only  in  a  limited  fashion. 

5.  The  type  of  contract  let  for  the  development  of  a 
software  product  has  a  definite  effect  on  the  resultant 
quality  associated  with  that  developed  product.  Specific¬ 
ally,  a  firmed  fixed  price  contract  appeared  to  result  in 
higher  quality  of  the  software  product,  according  to  several 
of  the  PMs  interviewed. 

6.  In  two  programs,  programs  one  and  seventeen,  to 
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assist  in  the  development  of  quality  software,  PMs  withheld 
progress  payments  when  software  development  was  not  progres¬ 
sing  in  a  satisfactory  manner.  The  withholding  of  progress 
payments  appeared  to  positively  contribute  to  the  overall 
quality  of  the  software  product. 

SQA  Discussion.  As  previously  mentioned,  this 
thesis  has  investigated  the  impact  of  the  degree  of  excel¬ 
lence  of  an  SQA  program  on  the  cost  and  schedule  integrity 
of  a  given  software  development  effort.  And,  while  the 
analysis  in  chapter  five  suggested  no  significant  relation¬ 
ship  between  the  degree  of  excellence  of  an  SQA  program  and 
the  cost  and  schedule  integrity  of  a  program,  what  is 
disturbing  is  the  large  number  of  programs  classified  as 
having  "unsound"  SQA  programs.  Consequently,  although  many 
factors  may  account  for  this  rating,  it  appears  that  a 
problem  does  currently  exist  in  the  area  of  SQA.  To  eval¬ 
uate  this  problem,  a  more  global  assessment  of  this  situa¬ 
tion  is  needed. 

In  the  SQA  literature  reviewed,  the  suggestion  was  made 
that  quality  software  was  obtained  through  an  objective, 
measurable  assessment  of  a  software  product  and  by  following 
managerial  procedures  through  the  course  of  the  software 
development  process.  DoD  efforts  to  build  quality  software 
presently  focus  on  following  managerial  procedures.  There¬ 
fore,  DoD  is  focusing  on  only  half  of  the  problem —  the 
procedural  development  of  software.  Uhat  is  also  needed 
is  to  objectively  define  software  quality  attributes. 
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Specifically,  DoD  needs  to  formally  define  software  quality 
as  a  total,  tangible  concept.  Currently,  in  all  facets  of 
software  development  within  the  program  office,  real  confu¬ 
sion  exists  with  regard  to  the  precise  meaning  of  SQA.  Lack 
of  clear  definition  of  SQA  and  an  apparent  lack  of  trained 
and  experienced  software  development  personnel  seem  to  be 
the  key  contributing  factors  to  the  continued  confusion 
associated  with  software  development. 

Baseline  Management  Discussion.  Like  SQA,  the  primary 
research  objective  concerning  baseline  management  was  to 
evaluate  its  effect  on  cost  and  schedule  integrity.  The 
statistical  inferences  were  discussed  in  chapter  five;  how¬ 
ever,  other  interesting  information  and  observations  were 
uncovered  in  the  course  of  interviewing  and  are  shared  here. 

1.  When  queried  as  to  what  guidance  was  used  in  estab¬ 
lishing  their  baseline  management  plans,  the  majority  of 
program  managers  stated  that  MIL-STD-483  and  MIL-STD-490 
were  used.  Neither  AFR  45-3  or  AFR  800-14  were  indicated  as 
the  reference  sources  in  any  of  the  interviews. 

The  accumulated  data  does  not  support  the  program 
managers'  claim  that  baseline  actions  are  in  accordance  with 
Military  Standard  483.  Further,  Mi  1 i tary  Standard  490  does 
not  provide  guidance  for  baseline  control.  Therefore,  this 
evidence  supports  the  observation,  reported  earlier,  that 
existing  documents  are  not  clear  in  their  guidance  or  devia¬ 
tions  are  being  taken  by  program  managers.  Additionally,  it 
is  important  to  note,  that  it  is  not  recognized,  by  program 


r. 


managers,  that  AFR  800-14  provides  the  guidance  which  should 
be  adhered  to. 

2.  Without  prompting,  seme  program  managers  explained 
why  the  allocated  baseline  was  delayed  beyond  PDR,  and,  in 
some  cases,  as  late  as  PCA/FCA  ,  According  to  these 
managers,  control  o-f  an  allocated  baseline  earlier  than 
FCA/PCA  is  too  early  and  will  certainly  result  in  costly 
engineering  change  proposals.  It  was  con-firmed  that 
development  specifications  ( B^)  were  reviewed  and  monitored, 
but  the  contractor  had  the  latitude  to  change  the  require¬ 
ments  in  the  specification  as  needed. 

Of  the  13  programs  which  deviated  from  the  baseline 
management  standard,  only  two  had  a  cost  overruns.  Thus,  it 
seems  that  the  claim  may  be  legitimate;  however,  it  was  not 
determined  if  the  delivered  product  met  original  performance 
requ i rements,  or,  because  of  the  limited  control  which  the 
government  had,  if  the  delivered  product  was  less  than 
satisfactory.  If  this  information  had  been  solicited,  a 
determination  could  have  been  made  as  to  the  effectiveness 
of  this  approach.  That  is,  while  a  cost  overrun  is  avoided, 
does  the  program  result  in  a  useable  product  which  fully 
satisfied  performance  requirements.  If  not,  would  an  ECP 
that  is  initiated  at  the  end  of  development  be  more  costly 
than  would  an  ECP  initiated  earlier  in  the  program,  and 
would  any  schedule  slip  incurred  now  have  been  avoided  by 
initiating  an  ECP  earlier? 
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3.  Surprisingly,  analysis  shows  the  effect  of 
following  the  standard  baseline  guidance,  to  have  a  negative 
consequence.  The  data  finds  four  of  the  five  programs  which 
followed  the  standard  to  have  schedule  slips.  In  reviewing 
other  program  data,  however,  two  alternative  explanations 
are  offered  for  this  unexpected  finding. 

First,  two  of  the  four  programs  had  established  base¬ 
lines  more  rigidly  than  the  standard  guidance  suggests. 

Thus,  while  following  the  standard  may  be  beneficial,  too 
rigid  an  application  may  have  an  adverse  impact,  as  the  data 
suggests. 

A  second,  more  plausible  explanation,  however,  may  be 
that  a  realistic  schedule  is  a  significant  moderating 
variable.  This  observation  is  based  on  the  fact  that  three 
of  these  four  programs  reported  not  having  a  realistic 
schedule  going  into  the  program.  Credence  for  this  explana¬ 
tion  is  also  provided  by  program  13.  It  is  the  only  program 
categorized  as  not  following  the  standard  which  has  a  sched¬ 
ule  slip,  but  it  also  reports  not  having  a  realistic  sched¬ 
ule  going  into  the  program. 

Other .  Other  unexpected  discoveries  were  made  in  the 
course  of  investigation.  While  not  specifically  related  to 
the  effect  that  SQA  or  baseline  management  have  on  cost  and 
schedule  integrity,  these  findings  suggest  provocative, 
alternate  reasons  which  may  account  for  program  perturba¬ 
tions  not  considered  by  the  researchers,  and  highlight 
interesting,  noteworthy  observations. 


82 


I 


1.  The  collected  data  disproves  what  the  literature 
suggests.  No  overwhelming  evidence  was  collected  which 
substantiates  the  criticism  of  software  as  being  the  'long 
pole'  or  as  a  major  expense  item  for  the  government.  Consi¬ 
deration  of  the  following,  however,  must  be  made.  First, 
the  data  collected  only  accounts  for  the  development  phase 
of  software.  Therefore,  while  it  appears  that  control  of 
cost  and  schedule  during  development  is  being  achieved, 
study  into  the  operation  and  maintenance  of  software  is 
needed  before  any  real  conclusion  can  be  drawn.  Second,  the 
programs  were  examined  only  by  cost  to  the  government.  In 
many  cases,  the  program  managers  stated  that  the  contractors 
were  experiencing  cost  overruns  not  chargeable  to  the 
government.  Thus,  a  study  of  industry  may  disclose  evidence 
which  supports  observations  that  software  is  a  management 
probl em. 

2.  If  a  program  does  not  have  realistic  budget,  sched¬ 
ule,  or  performance  specification,  then,  no  matter  how  good 
an  SGA  program  or  strong  the  baseline  control,  a  cost  over¬ 
run  and  schedule  sip  would  be  inevitable.  The  collected 
data  does  provide  good  evidence  that  this  premise  is  true 
for  schedule  slips  but  not  for  cost  overruns.  Of  the  six 
programs  used  in  statistical  analysis  and  reported  to  have  a 
schedule  slip,  five  were  also  identified  as  not  having  a 
realistic  ingoing  program  schedule.  Of  the  four  programs 
used  in  statistical  analysis  and  reported  to  have  a  cost 
overrun,  only  one  reported  having  an  unrealistic  budget 
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going  into  the  program.  However,  review  of  the  explanatory 
notes  <in  Appendix  8  o-f  chapter  five)  for  programs  4  and  9 
indicate  that  the  original  programs  were  not  properly  struc¬ 
tured  and  after  renegotiation  program  4  doubled  in  price  and 
program  9  received  one  tenth  of  the  originally  required 
products.  It  is  difficult  to  determine  a  reasonable  esti¬ 
mate  of  what  the  overrun  was  for  these  programs,  but  it  is 
obvious  that  substan t i al 1 y  mi 'e  was  paid  for  the  products 
finally  received  than  was  originally  planned  to  be  spent. 

Directions  for  Future  Research 

Through  the  course  of  this  research,  it  became  apparent 
that  the  area  of  SQA  and  baseline  management,  as  related  to 
software  development,  contained  many  possibilities  for 
future  research.  In  conclusion,  the  following  suggestions 
are  presented  as  topics  for  future  research: 

1.  Clearly,  the  next  step  is  to  study  what  happens  to 
software  during  the  operation  and  maintenance  phase.  If  the 
performance  of  the  developed  software  examined  in  this 
thesis  could  be  evaluated  out  in  the  field,  the  results 
could  be  related  to  the  development  characteristics  recorded 
here  and  long  term  effects  could  be  analyzed.  Such  an 
analysis  would  provide  important  insight  into  the  relation¬ 
ship  between  early  program  control  and  decisions  and  out 
year  effects  in  terms  of  life  cycle  cost  effectiveness.  How 
well  did  the  software  meet  performance  requirements  and  how 
maintainable  was  the  software? 


2.  While  this  thesis  effort  reviewed  and  presented  the 
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idea  that  SQA  was  a  two  component  -function  o-f  objectively 
assessing  so-ftware  quality  and  procedurally  managing  soft- 
ware  quality,  empirically  this  concept  was  only  partially 
explored.  Consequently,  it  is  suggested: 

a.  that  a  literature  review  and  progress  analysis 
be  undertaken  to  objectively  assess  past  and  present  DoD 
e-f-forts  to  incorporate  and  utilize  so-ftware  metrics. 

b.  that  programs  utilizing  so-ftware  metrics  be 
analyzed  in  terms  o-f  the  extent  o-f  metric  usage  and  the 
extent  o-f  user  satisfaction. 

c.  that  an  attempt  be  made  to  precisely  define 
what  is  meant  and  what  should  be  meant  by  the  term  “software 
quality"  within  the  DoD. 

3.  The  concept  of  quality,  as  presently  interpreted  by 
the  DoO,  needs  to  be  researched  in  terms  of  user  satisfac¬ 
tion.  Consequently,  analysis  related  to  the  degree  of 
excellence  of  an  SQA  program  during  software  development, 
training  and  experience  of  PMs  when  managing  software 
development  efforts,  and  the  contract  type  should  be 
conducted  to  determine  their  effect  on  user  satisfaction. 

4.  In  the  course  of  investigation,  the  researchers 
were  provided  an  incredible  example  of  how  market  structure 
may  affect  contractor  performance.  From  the  onset  of  this 
example  program,  the  contractor  failed  to  meet  schedule 
deadlines.  Although  a  strongly  worded  letter  of  admonish¬ 
ment,  by  the  ASD  commander,  was  sent  to  the  president  of  the 
company,  the  contractor's  performance  has  never  really 
improved.  The  program  manager  stated  that  the  contractor  is 
the  only  source  capable  of  working  the  technology  peculiar 
to  contract  and  for  this  reason  is  not  intimidated  or 
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concerned  by  customer  expressed  dissatisfaction.  This  is  an 
extreme  example,  but  it  does  provide  evidence  that  monopo¬ 
listic  power  can  prove  detrimental  to  a  program's  manage¬ 
ment.  Further  compounding  this  problem  is  the  decreasing 
number  of  vendors  available  and/or  willing  to  work  with  the 
government.  How  prevalent  these  problems  are,  and  their 
affect  on  cost  and  schedule  are  interesting  and  important 
subjects  -for  -future  study. 

5.  The  research  was  limited  to  analysis  ot  programs 
managed  at  ASD.  Study  is  required  o-f  the  other  Air  Force 
Systems  Command  Product  Divisions.  Repetition  o-f  this  pro¬ 
ject  would  contribute  towards  its  validity.  Additionally, 
it  may  uncover  other  important  -findings  not  revealed  here. 

6.  Unanimously,  PMs  agree  on  the  need  -for  a  relation¬ 
ship  or  working  interaction  between  the  disciplines  o-f  SGA 
and  baseline  management.  However,  when  PMs  were  asked  to 
indicate  the  context  of  the  relationship  and  potential  de¬ 
gree  o-f  interaction,  PMs  could  not  identi-fy  a  format  for 
establishing  such  a  working  interaction.  Consequently,  a 
need  exists  for  further  exploration  to  be  conducted  on  the 
topic  of  establishing  a  format  or  context  for  the  interaction 
of  SQA  and  baseline  management.  This  would  involve  defining 
the  current  role  of  SQA  as  it  is  both  perceived  and  formally 
specified  and  defining  the  role  of  baseline  management  as  it 
is  both  perceived  and  formally  specified,  and  then  exploring 
the  potential  of  combining  these  two  areas  and  the  resultant 
effects  on  the  cost,  schedule  and  quality  of  a  product. 
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7.  Lastly,  a  similar  research  project  should  examine 
the  contractor  side  o-f  the  software  development  business. 

As  pointed  out,  the  collected  data  did  not  support  claims 
that  so-ftware  is  a  cost  and  schedule  headache.  Because  o-f 
the  way  the  government  accounts  -for  performance,  especially 
in  a  -firm  -fixed  price  arrangement,  the  contractor's  cost 
overruns  are  not  considered  by  the  government  -for  reporting 
purposes.  Even  -for  cost  type  arrangements,  the  government 
shares  in  the  cost  overrun  only  up  to  the  ceiling,  price 
beyond  which  the  government  is  not  concerned,  -for  reporting 
purposes,  about  further  overruns  incurred  by  the  contractor. 

In  industry,  however,  these  overruns  are  recognized;  there¬ 
fore,  their  historical  cost  data  might  provide  evidence  more 
in  line  with  the  concerns  expressed  in  current  literature. 

Summary 

In  this  chapter,  the  overall  research  effort  was 
reviewed  and  general  observations  were  presented  as  derived 
from  the  research  effort.  While  it  could  not  be  statistically 
concluded  that  a  "Sound"  SQA  program  or  following  the  base¬ 
line  management  standard  influences  the  cost  effective 
development  of  quality  software,  it  is  hoped  that  the  obser¬ 
vations  recorded  will  provide  useful  management  insights  and 
stimulate  future  research.  Developing  quality  software, 
which  is  both  within  cost  and  on  schedule,  remains  a  signi¬ 
ficant  concern  within  the  DoD.  Therefore,  continued 
research  is  urgently  needed  to  address  this  problem. 
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SECT I CN  1;  BACKGROUND  INFORMATION 

1.  Uhat  type  o-f  contract  strategy  is  your  program  employ- 
i  ng? 

2.  What  price  was  the  contract  awarded  -for? 

2a.  Of  this  price,  how  much  is  -for  software  effort? 

3.  Would  you  please  tell  me  what  the  initial  PMRT  and  IOC 
dates  were  at  contract  award? 

3a.  Have  there  been  any  slips  to  these  dates? 

4.  Has  the  program  incurred  any  cost  overruns  due  to  soft¬ 
ware  probl ems? 

5.  Has  the  program  experienced  any  schedule  slips  due  to 
so-ftware  problems? 

6.  As  a  program  manager,  what  do  you  consider  a  significant 
cost  overrun? 

7.  As  a  program  manager,  what  do  you  consider  a  significant 
schedul e  slip? 


SECTION  II;  SOFTWARE  QUALITY  ASSURPMCE 


8.  Does  this  Program  Office  have  a  written  SGA  policy, 
which  was  either  developed  by  you,  the  PM,  or  seme  higher 
level,  emphasizing  the  overall  management  objectives  of  a 
SQA  Program? 

9.  Does  the  contractor  on  this  program  have  a  written  SQA 
policy  emphasizing  the  overall  management  objectives  of  a 
SQA  Program? 

10.  If  such  a  contractor  SQA  policy  existed,  was  it 
reviewed  by  Program  Office  personnel? 

11.  For  this  program,  did  the  contractor  have  a  specific 
SQA  pi  an? 

12.  If  such  a  plan  existed,  were  standards  established 
for  documents,  test  coverages,  and  configuration  management 
reports  in  the  plan? 


88 


13.  If  such  a  plan  existed,  were  test  tools  and  test  sup¬ 
port  so-ftware  outlined  in  the  plan? 

14.  I-f  such  a  plan  existed,  was  a  sequence  of  formal  and 
possibly  informal  reviews  outlined  in  the  plan  which  would 
delineate  the  progression  of  software  development? 

15.  Does  the  contract  for  this  program  specify  the  use  of 
MIL-S-52779A  for  software  quality  assurance  requirements? 


Independent  Support. 

16.  On  this  program  was  an  Independent  Verification  and 
Validation  <IV&V>  team  utilized? 

17.  On  this  program  did  the  contractor  have  an  internal 
quality  control  section  for  this  development  effort? 

18.  If  such  a  section  exists,  has  the  contractor 
documented  a  management  plan  to  periodically  assess  the 
effectiveness  of  this  section? 

19.  Does  the  contractor  have  established  a  software 
discrepancy  reporting  system  to  assure  that  all  known  prob¬ 
lems  are  documented  and  tracked  for  corrective  action? 


Test i no. 


20.  Does  the  contractor's  attitude  seem  to  emphasize  the 
early  detection  of  deficiencies  in  the  software  development 
process? 

21.  Does  the  SQA  plan  identify  the  contractor's  software 
test  activities,  and  if  so,  does  the  SQA  plan  allow  these 
activities  to  be  monitored  for  the  certification  that  test 
results  are  the  actual  findings  of  the  test  activities? 

22.  Do  you  recall  the  test  tools  used  in  software  testing? 

23.  Were  Software  Metrics  used  in  software  testing? 


Software  Development  Progression. 

24.  In  analyzing  the  developing  software  product 
throughout  the  software  development  process,  was  software 
documentation  developed  and  maintained  to  your 

sat i sf ac  t i on? 

25.  Were  formal  and  possibly  informal  reviews  conducted  to 
monitor  the  software  development  process? 
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26.  Was  a  successful  Preliminary  Design  Review  conducted 
•for  each  Configuration  I  tern  before  proceeding  into  detailed 
de  sign? 

27.  Uas  a  successful  Critical  Design  Review  conducted  for 
each  Configuration  Item  prior  to  coding? 

28.  Before  a  new  phase  of  software  development  was 
undertaken,  was  work  in  the  preceding  phases  of  software 
development  accomplished  to  your  satisfaction? 


SECTION  III;  BASELINE  MANAGEMENT 


Procjram  Baseline  Management  Strategy. 


29.  Was  there  a  baseline  management  policy  established 
prior  to  contract  award? 


30.  Was  this  policy  adhered  to? 


31.  If  yes,  is  there  a  document  you  can  furnish  us  that 
discusses  this  policy? 


31a.  Was  this  policy  based  on  any  particular  MIL-STD 
or  AFR? 


Prooram  Baseline  Management  Actions. 


32.  Uhen  was  functional  baseline  established? 


33.  When  was  allocated  baseline  established? 


34.  At  what  point  did  the  contractor  establish  an  internal 
product  baseline? 


SECTION  IV;  OTHER 

35.  Do  you  feel  that  the  ingoing  program  had  a  realistic 
budget,  schedule,  and  system  specification? 

36.  Based  on  your  experience,  do  you  see  a  relationship 
(or  the  need  for  a  defined  relationship)  between  SQA  and 
baseline  management?  If  so,  could  you  elaborate  please? 
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Appendix  B:  Program  Data  Summary 


Program  1 

Contract  Type:  CPIF  Program  Cost:  $106M 

Software  Cost:  $35M  Cost  Overrun:  $4M 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  1 5X 

Significant  Schedule  Slip:  3  months  beyond  delivery  date 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  The  PM  stated  that  upper  level 
management  support  existed  behind  both  AF  and  contractor 
efforts  on  this  program.  However,  it  is  questionable  as  to 
the  effectiveness  of  this  emphasis.  The  PM  indicated  that 
while  an  SGA  plan  existed,  the  plan  was  more  of  a  show  item. 
Further,  it  was  discovered  that  only  a  cursory  review  of  the 
SQA  plan  was  conducted.  Lastly,  the  PM  indicated  that  while 
the  standard  formal  and  informal  reviews  and  audits  were 
conducted,  in  the  opinion  of  the  PM,  these  reviews  and 
audits  were  not  wholly  effective  due  to  a  'build  then  fix* 
attitude  of  the  contractor. 

B.  Independent  support:  A  contracted  Independent 
Verification  and  Validation  <IV&V>  team  was  used  on  this 
program,  and  in  the  opinion  of  the  PM  was  effective  in  the 
identification  of  software  problems.  The  contractor  did 
have  an  independent  SQA  function,  but  the  effectiveness  of 
this  section  could  not  be  determined. 
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C.  Testing:  Extensive  testing  was  conducted  by  the  AF 


and  as  a  result  of  this  testing  numerous  software  errors 
were  detected.  In  the  opinion  of  the  FM,  the  contractor 
developing  this  software  product  did  not  possess  an  attitude 
relating  to  the  early  identification  of  errors.  Conse¬ 
quently,  since  testing  was  conducted  in  the  latter  stages  of 
software  development,  detected  software  errors  were  more 
costly  to  correct. 

D.  Progression:  As  stated  in  the  Management  support 
section  of  this  analysis,  according  to  the  PM,  the  progres¬ 
sion  of  the  software  development  process  was  unsatisfactory. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MIL— S— 52779A:  Used. 

3.  Although  this  program  is  classified  as  posses¬ 
sing  an  unsound  SGA  program,  the  PM  stated  that 
a  quality  software  product  was  being  produced. 
This  appears  to  be  the  result  of  the  PM 
withholding  payments  through  the  course  of  the 
software  development  effort  until  the  software 
produced  actually  performed  as  required.  It 
should  be  noted  that  as  this  practice  was 
utilized  by  the  PM,  in  the  opinion  of  the  PM, 
contractor  upper  level  management  support  became 
more  evident  as  evidenced  by  an  apparent  shift 
in  the  contractor's  attitude  toward  the  early 
detection  of  errors  and  a  smoother  software 
development  progression. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 

Allocated  baseline  was  delayed  till  past  CDR;  thus,  the 
program  fails  to  meet  the  criteria  of  the  baseline  manage¬ 
ment  standard. 
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Program  2 

Contract  Type:  FFP  Program  Cost:  S4.2M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  SO 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  No  Response 
Significant  Schedule  Slip:  No  Response 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION:  NCNE ■ 

Neither  the  PH  nor  the  lead  software  engineer  on  the 
program  choose  to  respond  to  the  SQA  portion  of  the  inter¬ 
view.  This  program  was  not  included  in  chapter  five  SQA 
cal cul at i ons . 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 

According  to  the  program  manager,  because  this  is  an 
FFP  contract,  it  is  not  to  the  program's  benefit  to  estab¬ 
lish  baselines.  The  contractor  should  be  left  to  manage  his 
own  baseline.  The  program  office  will  establish  baseline 
control  at  PCA.  Thus,  this  program  is  not  considered  to 
meet  the  requirements  of  the  baseline  management  standard. 

Program  3 

Contract  Type:  FFP  Program  Cost:  S4.9M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  $0 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  No  Response 
Significant  Schedule  Slip:  No  Response 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 
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SQA  CLASSIFICATION;  NONE 


Neither  the  PM  nor  the  lead  software  engineer  on  the 
program  choose  to  respond  to  the  SQA  portion  o-f  the  inter¬ 
view.  This  program  was  not  included  in  chapter  -five  SQA 
cal cul at i ons. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 

Allocated  baseline  is  being  delayed  till  FCA/PCA;  thus, 
this  program  does  not  meet  the  criteria  o-f  the  baseline 
management  standard. 

Program  4 

Contract  Type:  FFP  Program  Cost:  S200M 

So-ftware  Cost:  Data  Unavailable  Cost  Overrun:  See  Note 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  5X 

Significant  Schedule  Slip:  When  the  mission  cannot  be  met. 

Realistic  Budget:  No 

Realistic  Schedule:  No 

Realistic  System  Specification:  Yes 

Note:  This  program  was  originally  structured  as  a 

S91M  FPI  contract.  One  year  later,  with  congressional 
approval,  the  program  was  renegotiated  as  an  FFP  contract 
for  $200M.  Uih  i  1  e  the  program  doubled  in  price  the  scope  of 
effort  remained  virtually  unchanged.  Because  of  the  signi¬ 
ficant  change  in  price  this  program  is  considered  an  excep¬ 
tional  deviation  and  disregarded  for  statistical  testing. 
However,  it  does  provide  a  good  example  of  a  program  with  an 
initially  unrealistic  schedule  and  budget. 
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SQA  CLASSIFICATION:  SOUND. 

A.  Management  support:  Both  the  AF  and  the  contractor 
appeared  to  have  upper  level  management  support  emphasizing 
the  significance  of  SQA.  An  SQA  plan  was  in  existence  and, 
in  the  opinion  of  the  PM,  was  thoroughly  reviewed  in  a  joint 
effort  by  both  program  office  and  contractor  personnel.  The 
PM  indicated  that,  through  the  course  of  the  program,  the 
SQA  plan  was  constantly  under  revision  pursuant  to  the 
intent  of  a  workable  SQA  program.  The  SQA  plan  included 
such  items  as  the  establishment  of  documentation  standards, 
outlining  of  needed  software  testing  tools  and  support  soft¬ 
ware,  and  the  methodology  of  the  software  development  pro¬ 
gress  i  on  . 

B.  Independent  support:  Initially,  neither  a 
contracted  nor  a  Program  Office  IV&V  team  was  utilized; 
however,  after  the  PM  recognized  a  lack  of  experienced  AF 
personnel  to  read,  test,  detect  and  interpret  software  prob¬ 
lems  within  the  Program  Office,  a  contracted  IV&V  team  was 
employed.  The  PM  held  that  the  IUScV  team  utilized  signifi¬ 
cantly  contributed  to  the  success  of  the  software  product. 
Also,  the  contractor  did  have  an  internal  SQA  section,  which 
the  PM  classified  as  effective.  The  contractrr's  SQA  sec¬ 
tion  had  a  direct  link  with  the  company  president  as  well  as 
having  such  items  as  a  software  discrepancy  reporting  system 
to  document  and  track  software  problems  for  corrective 
action.  In  addition,  the  contractor  had  an  SQA  section 
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inspection  program  to  periodically  assess  the  e-f  -f  ec  t  i  veness 
o-f  the  SGA  section. 


C.  Testing:  All  software  was  tested  by  the  contractor 
at  least  twice  before  testing  was  actually  monitored  by  A F 
personnel.  Further,  in  the  opinion  o-f  the  PM  the 
contractors  attitude  seemed  to  be  one  o-f  emphasizing  the 
early  detection  o-f  errors. 

D.  Progression:  So-ftware  development  -followed  an 
orderly  progression,  which  was  to  the  sat  i  s-f  ac  t  i  on  o-f  the 
PM. 


E.  Other: 

1.  So-ftware  Metrics:  Used. 

2.  MIL-S-52779A:  Used. 

3.  This  program  seemed  to  have  one  o-f  the  best 
SGA  programs  with  two  items  standing  out  as 
significantly  contributing  to  the  success  o-f 
this  program: 

a.  The  PM's  recognition  of  the  significance 
of  SQA  and  determination  to  implement  an 
SQA  program. 

b.  The  PM  was  able  to  develop  a  sound  working 
relationship  with  the  contractor. 

c.  The  contract  type  appeared  to  force  the 
contractor  to  work  within  cost  and  schedule 
requ i remen  ts. 


MEETS  EASELINE  MANAGEMENT  STANDARD:  NO. 

The  allocated  baseline  is  being  delayed  till  FCA; 
therefore,  it  does  not  meet  the  baseline  management  stand¬ 
ard's  requirements. 
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Program  5 

Contract  Type:  FFP  Program  Cost:  $16.2M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  SO 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  10'/. 

Significant  Schedule  Slip:  Not  until  someone  else  is  being 

seriously  impacted. 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SQA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  For  this  program,  it  appeared 
that  there  was  not  any  AF  upper  level  management  support  and 
even  the  PM  did  not  appear  to  emphasize  the  significance  of 
SGA.  The  PM  was  not  aware  of  any  upper  level  management 
emphasis  concerning  SQA  by  the  contractor.  An  SQA  plan  was 
not  specifically  developed  for  this  program,  but  instead 
developed  within  the  context  of  the  Configuration  Management 
plan. 

B.  Independent  support:  Neither  a  contracted  nor  a 
Program  Office  IV&V  team  was  used,  but  the  contractor  did 
have  an  independent  SGA  section.  The  PM  did  not  make  a 
determination  as  to  the  effectiveness  of  the  contractor's 
SQA  section,  although  the  PM  did  state  that  a  software 
discrepancy  reporting  system  existed  and  appeared  effective. 

C.  Testing:  All  developed  software  was  tested  several 
times  before  presentation  to  AF  personnel  for  testing.  The 
PM  stated  that  the  contractor  emphasized  the  early  detection 
of  software  errors  during  development. 
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D.  Progression:  All  -formal  and  in-formal  reviews  and 
audits  -followed  an  orderly  progression,  which  was  to  the 
satisfaction  of  the  PM. 


E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MIL-S-52779A:  Not  used. 

3.  This  particular  PM  did  not  view  SQA  as  overly 
significant.  Instead,  the  PM  emphasized  the 
firm  fixed  price  type  of  contract  as  the 
driving  factor  in  the  development  of  a  quality 
software  product. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 


The  allocated  baseline  was  established  after  CDR;  thus, 
it  does  not  meet  the  criteria  of  the  baseline  management 
standard. 


Program  6 


Contract  Type:  CPIF 
Software  Cost:  S2GM 
Schedule  Slip:  0  months 


Program  Cost:  $100M 
Cost  Overrun:  S1.44M 


Significant  Cost  Overrun:  25X 

Significant  Schedule  Slip:  3  months  after  ICC. 


Realistic  Budget:  Yes 

Realistic  Schedule:  No 

Realistic  System  Specification:  Yes 


SQA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  Uithin  the  AF ,  upper  level 
management  support  did  not  exist.  However,  the  contractor 
did  have  an  upper  level  management  policy  emphasizing  the 
significance  of  SGA.  Further,  the  PM  believed  the  intent  of 
this  policy  guidance  was  adhered  to  by  the  contractor 
throughout  the  software  development  process,  with  the  excep- 
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t  i  on  of  the  contractor's  attitude  toward  early  detection  o-f 
errors.  An  SGA  plan  was  developed  tor  this  program;  how¬ 
ever,  it  was  not  reviewed  by  Program  Office  personnel. 

B.  Independent  support:  Initially,  neither  a 

contracted  nor  a  Program  Office  team  was  used,  but, 

after  the  PM  recognized  the  lack  o-f  expertise  in  the  area  o-f 
software  development  within  the  Program  Office,  a  contracted 
IV&V  team  was  utilized.  The  contractor  did  have  an 
independent  SGA  section  which  he  believed  was  e-f-f  ec  t  i  vel  y 
contributing  to  the  software  development  e-f-fort.  Quality 
documentation  accompanying  the  software  develooed  substan¬ 
tiated  the  FM's  claim. 

C.  Testing:  Throughout  the  testing  process,  the  atti¬ 
tude  of  the  contractor  did  not  seem  to  emphasize  the  early 
detection  of  errors. 

D.  Progression:  In  the  opinion  of  the  PM,  before  a 
new  phase  of  software  development  was  undertaken,  work  in 
the  preceding  phases  had  not  been  satisfactorily  accomp¬ 
lished.  Software  development  did  not  follow  an  orderly 
progress! on . 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MI L-S-52779A :  Not  used. 

3.  The  PM  did  not  appear  knowledgeable  on  the 
topic  of  SQA  or  even  of  the  arguments  either 
for  or  against  the  significance  of  SGA.  And, 
while  the  Program  Office  lead  engineer  was  more 
knowledgeable  of  the  SQA  subject  area,  the 
complaint  was  that,  due  to  a  lack  of  time  and 
manpower,  it  was  exceptionally  difficult  to  be 
thoroughly  concerned  with  SQA  for  this  particu¬ 
lar  program. 
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MEETS  BASELINE  M£MAGEMENT  STANDARD ;  NO. 

The  allocated  baseline  was  delayed  beyond  PDR;  there- 
-fore,  it  does  meet  the  criteria  of  the  baseline  management 
standard. 

Program  7 

Contract  Type:  FPIF  Program  Cost:  *29M 

So-ftware  Cost:  ^14.5M  Cost  Overrun:  SO 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  20% 

Significant  Schedule  Slip:  20%  of  contract  period  beyond 
IOC. 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SQA  CLASSIFICATION:  SOUND. 

A.  Management  support:  With  the  exception  of  the  PM, 

A F  upper  level  management  support  did  not  apparently  exist 
on  this  program.  The  PM  stated  he  was  aware  of  upper  level 
management  support  by  the  contractor.  An  SQA  plan  did  exist 
for  this  software  development  effort,  and  the  SQA  plan  was 
reviewed  by  Program  Office  personnel.  The  SQA  plan  did 
establish  documentation  standards,  outline  test  tools  and 
test  support  software  and  delineate  the  software  development 
progress i on . 

B.  Independent  support:  A  contracted  IV&V  team  was 
used,  and  the  contractor  did  have  an  independent  SQA  sec¬ 
tion.  Further,  the  contractor  did  utilize,  in  the  opinion 
of  the  PM,  an  effective  software  discrepancy  reporting 
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system  to  document  and  track  corrective  actions  on  identi¬ 


fied  software  discrepancies.  The  contractor  also  had  a 
program  to  periodically  assess  the  effectiveness  of  the  SGA 

sec  t  i  on . 

C.  Testing:  The  contractor  tested  the  software 
several  times  prior  to  A F  monitoring  of  tests.  The 
contractor's  attitude  seemed  to  emphasize  the  early  detec¬ 
tion  of  errors. 

D.  Progression:  The  PM  was  satisfied  that,  before  a 
new  phase  of  software  development  was  undertaken,  work  in 
preceding  phases  was  accomplished  to  his  sat i sf ac t i on .  The 
PM  stated  that  all  formal  and  informal  reviews  conducted 
were  successful  and  software  development  did  follow  an 
orderly  progression. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MIL-S-52779A:  Not  used. 

3.  Although  SGA  practices  appeared  to  contribute  to  the 
development  of  a  quality  software  product,  several 
other  factors  seemed  to  contribute  to  the  success 

of  this  program. 

a.  The  PM  appeared  a  har dwork i ng ,  highly 
motivated  individual  who  recognized  the 
potential  of  SQA. 

b.  The  PM  insisted  on  a  realistic  budget  and 
schedul e . 

c.  The  type  of  contract  appeared  to  force  the 
contractor  to  produce  a  quality  software 
product . 

MEETS  BASELINE  MANAGEMENT  STANDARD :  NO. 

The  allocated  baseline  was  established  at  CDR;  there- 
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fore,  the  program  does  not  meet  the  criteria  of  the  baseline 
management  standard. 


Program  8 

Contract  Type:  FFP  Program  Cost:  $14M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  $0 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  20K  for  this  program 
Significant  Schedule  Slip:  1  month  beyond  delivery  date 

Realistic  Budget:  No 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  The  PM  stated  that  there  was  a 
growing  concern  on  the  part  of  upper  management  in  both  the 
AF  and  by  the  contractor  relating  to  SQA.  The  PM  stated 
that,  as  a  result  of  this  concern,  his  awareness  concerning 
SQA  had  increased.  An  SQA  plan  did  exist  and  was  reviewed 
by  Program  Office  personnel .  However,  the  PM  was  unaware  of 
the  contents  of  the  plan. 

B.  Independent  support:  Neither  a  contracted  nor  a 
Program  Office  IVAcV  team  was  used  on  this  program.  The 
contractor  did  have  an  independent  SQA  section,  but  the  PM 
could  not  comment  on  the  effectiveness  of  this  section. 

C.  Testing:  Initially,  the  contractor  appeared  to 
have  a  'build  then  fix'  attitude  concerning  software 
development.  However,  according  to  the  PM,  as  a  result  of 
the  increased  concern  of  AF  upper  level  management,  the 
contractor's  attitude  seemed  to  shift  to  one  of  detecting 
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errors  as  soon  as  possible  in  the  software  development 
ef -for  t . 

D.  Progression:  A-fter  the  expressed  increase  in  con¬ 
cern  -for  SQA  by  both  AF  and  contractor  upper  management,  all 
-formal  and  in-formal  reviews  and  audits  -followed  a  more 
orderly  progression,  which  was  to  the  sat  i  s-f  ac  t  i  on  o-f  the 
PM. 

E.  Other: 

1.  So-ftware  Metrics:  N^t  used. 

2.  MI L-S-52779A :  Used. 

3.  While  the  overall  quality  o-f  the  so-ftware 
product  being  produced  appeared  to  be  improv¬ 
ing,  two  -factors  other  than  SQA  seemed  to  be 
the  driving  -forces. 

a.  The  type  o-f  contract  appeared  to  forcing 
the  contractor  to  produce  a  higher  quality 
software  product. 

b.  Upper  level  management  support  from  both 
the  AF  and  contractor  appeared  to  be  con¬ 
tributing  to  the  development  of  a  higher 
quality  software  product. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  YES . 

The  functional  baseline  was  established  at  contract 
award,  allocated  baseline  at  PDR  and  product  baseline  at 
PCA,  the  program  therefore,  adhered  to  the  baseline  manage¬ 
ment  standard. 


Program  9 

Contract  Type:  CPIF  Program  Cost:  $21M 

Software  Cost:  $14. 5M  Cost  Overrun:  See  Note 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  1571 

Significant  Schedule  Slip:  3  months  beyond  delivery  date 
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Realistic  Budget:  No 

Realistic  Schedule:  No 

Realistic  System  Specification:  No 

*  NOTE:  Because  of  seme  significant  program  peculiarities, 
this  program  is  disregarded  for  statistical  testing.  This 
program  involved  new  technology  requiring  the  development  of 
sophisticated  software.  The  original  -$21M  program  was  let 
to  procure  ten  systems.  The  risk  was  not  properly  assessed 
and  two  years  after  the  initial  contract  was  awarded,  the 
effort  was  significantly  reduced,  and  only  one  system  was  to 
be  delivered  with  the  contractor  incurring  any  costs  beyond 
the  $21M.  Additionally,  no  reprocurement  data  was  procured, 
as  new  pol i cy  requires  contract  support  be  obtained  for  the 
type  of  software  being  acquired. 

SGA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  AF  upper  level  management 
support  did  not  overtly  exist.  With  regard  to  the 
contractor,  initially  upper  level  management  support  also 
did  not  exist.  However,  due  to  the  large  dollar  amounts  and 
the  type  of  contract  covering  this  program  as  combined  with 
continued  problems  experienced  early  in  the  program,  the 
contractor  changed  management  personnel.  Consequently,  with 
this  personnel  change,  upper  level  contractor  management 
support  of  SQA  practices  improved.  It  should  also  be  noted 
that  an  SQA  plan  did  exist  for  this  effort,  but  the  PM  was 
unaware  of  its  contents. 
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B.  Independent  support:  Neither  a  contracted  nor  a 


Program  Office  IV&V  team  was  used  on  this  program.  However, 
in  the  opinion  of  the  PM,  the  contractor  did  have  an  effect¬ 
ive  independent  SQA  section. 

C.  Testing:  After  the  change  in  management  personnel , 
testing  practices  were  revised  so  software  was  tested 
several  times  prior  to  the  monitoring  of  testing  activities 
by  the  AF.  Further,  the  contractor's  attitude  seemed  to 
shift  to  the  early  detection  of  software  d i scr epanc i es . 

D.  Progression:  After  the  change  in  management 
personnel,  software  development  followed  a  more  orderly 
progression  than  had  previously  been  experienced.  Specifi¬ 
cally,  before  a  new  phase  of  software  development  was  under¬ 
taken,  work  in  the  preceding  phases  was  accomplished  to  the 
satisfaction  of  the  PM. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MI L— S— 52779A :  Used. 

3.  A  comprehensive  SGA  program  did  not  appear  to 
exist  on  this  program.  Instead  ,the  success 
of  this  program  seemed  to  be  driven  by  other 

f  ac  tor  s . 

a.  The  type  of  contract  covering  this  program 
appeared  to  help  the  contractor  real¬ 
ize  that  it  was  in  his  own  best  interests 
to  utilize  the  potential  of  SQA  to  assist 
in  reducing  software  development  costs  and 
ultimately  produce  a  higher  quality  soft¬ 
ware  product. 

b.  The  change  in  contractor  management,  which 
resulted  in  increased  upper  level  manage¬ 
ment  support  by  the  contractor,  also 
appeared  to  contribute  to  the  production 
of  a  higher  quality  software  product. 
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MEETS  BASELINE  MANAGEMENT  STANDARD ;  NOT  APPLICABLE. 
No  specifications  were  being  procured. 


Program  10 

Contract  Type:  FFP  Program  Cost:  S31M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  $0 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  2X 

Significant  Schedule  Slip:  It  depends  on  the  program. 
Realistic  Budget:  Yes 

Realistic  Schedule:  Optimistic  schedule  (not  because  of 

sof tware) 

Realistic  System  Specification:  Yes 

Note:  This  program  required  the  procurement  of  test 

programs.  The  strategy  used  was  to  procure  factory  test 
programs.  Thus,  no  software  development  was  managed  under 
this  contract.  Because  of  this  peculiarity,  this  program  is 
disregarded  for  statistical  analyses. 

SQA  CLASSIFICATION :  NOT  APPLICABLE. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NOT  APPLICABLE. 


Program  11 

Contract  Type:  FFP  Program  Cost:  M5SM 

Software  Cost:  MOM  Cost  Overrun:  $0 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  No  response 
Significant  Schedule  Slip:  No  response 

Realistic  Budget:  No 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SQA  CLASSIFICATION;  UNSQLMD. 

A.  Management  support:  With  the  exception  of  the  PM, 
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upper  level  management  support  from  the  AF  did  not  exist. 

The  contractor  did  have  upper  level  management  policies 
emphasizing  the  significance  of  effective  SGA  practices. 
However,  through  the  course  of  the  interview  with  the  PM,  it 
was  difficult  to  determine  the  effectiveness  of  upper  level 
contractor  management  policies  towards  the  production  of  a 
quality  software  product.  Nevertheless,  an  SQA  plan  was 
developed  and  tailored  to  this  development  effort,  and  the 
SGA  plan  was  reviewed  by  Program  Office  personnel .  The  plan 
did  include  such  items  as  the  establishment  of  standards  for 
documentation,  the  tools  and  support  software  to  be  used  in 
testing  developed  software,  and  the  sequence  of  reviews  and 
audits  to  be  conducted. 

8.  Independent  supports  A  contracted  IV&V  team  was 
used  on  this  software  development  effort,  and  in  the  opinion 
of  the  PM  was  effective.  The  contractor  also  had  an 
independent  SGA  section;  however,  the  FM  could  not  comment 
on  the  effectiveness  of  this  section. 

C.  Testing:  All  tests  were  not  accomplished  to  the 
satisfaction  of  the  PM.  The  PM  noted  significant  problems 
were  experienced  in  the  area  of  integration  of  the  software 
and  hardware.  While  the  contractors  attitude  appeared  to 
emphasize  the  early  detection  of  software  discrepancies, 
this  software/  hardware  integration  problem  appeared  to 
detract  from  the  overall  success  of  this  program. 

D.  Progression:  The  PM  felt  that  the  software 
developed  for  this  program  was  taken  too  lightly  by  the 
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contractor.  Consequently,  the  PM  stated  that  although  the 
provisions  o-f  the  contract  were  being  -fulfilled,  software 
development  was  not  following  an  orderly  progression  and 
work  was  not  being  sat i sf ac tor i 1 y  acccmp 1 i shed. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MI L-S-52779A :  Not  used. 

3.  The  type  of  contract  appeared  to  have  more  of 
an  influence  on  the  development  of  the  software 
associated  with  this  program  than  the  attempts 
at  implementing  an  SQA  program. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 


Establishment  of  the  allocated  baseline  was  delayed 
till  FCA/PCA.  Thus,  the  program  fails  to  meet  the  require¬ 
ments  of  baseline  management  standard. 


Program  12 

Contract  Type:  FPIF  Program  Cost:  $39M 

Software  Cost:  $19. 5M  Cost  Overrun:  $0 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  3 0'/. 

Significant  Schedule  Slip:  2  weeks 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION:  SOUND. 

A.  Management  support:  From  an  AF  perspective,  upoer 
level  management  support  did  not  exist  concerning  SQA.  How¬ 
ever,  the  contractor  on  this  program  did  have  a  written  SGA 
policy  which,  according  to  the  PM,  did  highlight  the  signi¬ 
ficance  of  developing  quality  software.  An  SQA  plan  was 
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developed  specifically  for  this  program  and  was  reviewed  in 
detail  by  the  PM.  The  plan  did  contain  such  items  as  docu¬ 
mentation  standards,  tools  and  support  software  needed  for 
testing  and  the  formal  and  informal  reviews  to  be  conducted 
through  the  course  of  the  software  development. 

B.  Independent  support:  A  contracted  IV&V  team  was 
used  and  was  effective.  Also,  the  contractor  did  have  a 
section  specifically  dedicated  to  SGA,  which  the  PM  also 
classified  as  effective. 

C.  Testing:  All  software  developed  was  tested  several 
times  prior  to  being  tested  with  AF  review.  The  PM  believed 
that  the  contractor  strongly  emphasized  the  early  detection 
and  correction  of  errors.  The  PM  felt  that  the  contractor 
had  recognized  that  it  was  in  the  best  interest  of  the 
company  to  correct  errors  early. 

D.  Progression:  All  formal  and  informal  reviews  and 
audits  were  conducted  to  the  satisfaction  of  the  PM.  Soft¬ 
ware  development  followed  an  orderly  progression.  The  PM 
stated  that  before  a  new  phase  of  software  development  was 
undertaken  all  work  in  preceding  phases  was  accomplished  in 
a  satisfactory  manner. 

E.  Other: 

1.  Software  Metrics:  Metrics  were  used  by  the 

IV&V  contractor. 

2.  MI L-S-52779A :  Not  used. 

3.  Although  upper  level  AF  management  support  did 
not  overtly  exist  for  this  program,  several 
other  factors  strongly  contributed  to  this 
program. 
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a.  The  type  of  contract  seemed  to  help 
insure  that  a  quality  software  product 
was  being  produced. 

b.  Before  this  program  started,  the  PM 
insisted  on  a  realistic  budget  and 
schedu 1 e . 


c.  The  PM  had  nineteen  years  experience  in 
the  area  of  program  management  and 
appeared  to  recognize  the  problems  typi¬ 
cally  associated  with  software  development. 

MEETS  BASELINE  MANAGEMENT  STANDARD;  NO. 

The  establishment  of  allocated  baseline  was  at  FCA/PCA . 
Thus,  the  program  fails  to  meet  the  criteria  of  the  baseline 
management  standard. 


Program  13 

Contract  Type:  FFP  Program  Cost:  $2M 

Software  Cost:  Data  Unavailable  Cost  Overrun:  SO 

Schedule  Slip:  7  months 

Significant  Cost  Overrun:  20’/. 

Significant  Schedule  Slip:  25X  of  contract  schedule  beyond 

de 1 i very  date 

Realistic  Budget:  Yes 
Realistic  Schedule:  No 
Realistic  System  Specification:  Yes 

SSA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  AF  upper  level  management  sup¬ 
port  did  not  exist  to  assist  the  PM.  The  contractor  did  at 
least  verbally  espouse  the  significance  of  SQA.  However, 
according  to  the  PM,  contractor  emphasis  was  ineffective  in 
implementing  any  form  of  an  SGA  program.  The  PM  noted  that 
a  significant  problem  existed  within  the  contractors  plant; 
although  an  independent  SQA  function  existed,  this  function 
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was  newly  created  and  in  strong  conflict  with  the 
contractors  engineering  -function  which  previously  had  been 
responsible  for  quality  assurance  efforts.  It  should  be 
noted  that  an  SQA  plan  did  not  exist  -for  this  program  and 

that  the  documentation  produced  to  accompany  the  developed 
so-ftware  was  generally  unsatisfactory. 

8.  Independent  support:  Neither  a  contracted  nor 
Program  Office  IVScV  team  was  used.  And,  although  the  con¬ 
tractor  did  have  an  independent  SQA  section,  -for  reasons 
previously  mentioned,  this  section  was  classified  by  the  FM 
as  i  ne-f-f  ec  t  i  ve .  In  fact,  the  FM  believed  the  existence  o f 
the  contractors  SQA  section  and  the  constant  conflict  with 
the  contractors  engineering  section  actually  detracted  from 
the  program. 

C.  Testing:  The  contractor's  attitude  was  one  of 
"build  then  fix"  the  software. 

D.  Progression:  Although  formal  reviews  were  accepted 
as  complying  with  contractual  requirements,  the  quality  of 
the  software  product  being  produced  was  not  to  the  satisfac¬ 
tion  of  the  PM.  According  to  the  PM,  the  contractor  tried 
to  do  many  things  at  the  same  time  and  consequently  hardware 
integration  problems  resulted.  The  PM  concluded  that  soft¬ 
ware  development  efforts  did  not  follow  an  orderly  prcgree- 
s  i  on . 

E.  Other: 

1.  Software  Metrics:  Not  used. 


2.  MI L-S-52779A :  Not  used. 


3.  The  PM  did  not  seem  all  that  knowledgeable  on 
the  topic  o-f  software  development.  Further, 
the  type  of  contract  seemed  to  drive  any 
quality  inherent  in  the  software  product. 

MEETS  BASELINE  MANAGEMENT  STANDARD;  NO. 


The  allocated  baseline  was  established  at  PCA.  Thus, 
the  program  does  not  meet  the  requirements  of  the  baseline 
management  standard. 


Program  14 

Contract  Type:  FPI  Program  Cost:  $3£M 

Software  Cost:  $3M  Cost  Overrun:  $**00;< 

Schedule  Slip:  0  months 

Significant  Cost  Overrun:  15>£ 

Significant  Schedule  Slip:  3  months  beyond  delivery  elate 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION;  SOUND. 

A.  Management  support:  Upper  level  management  support 
existed  for  both  the  AF  and  contractor.  For  the  AF,  verbal 
emphasis  existed.  The  contractor  on  this  program  actually 
had  a  written  SGA  policy  which  effectively  conveyed  the 
significance  of  developing  quality  software.  This  program 
did  have  an  SQA  plan  which  was  reviewed  by  Program  Office 
personnel  and  deemed  sat i sf ac tory . 

B.  Independent  support:  According  to  the  PM,  a  con¬ 
tracted  IV&V  team  was  used  effectively  on  this  effort.  The 
contractor  did  have  an  independent  SQA  section,  which  the  PM 
believed  was  effectively  contributing  to  software  develop¬ 
ment  . 
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C.  Testing:  All  software  was  tested  several  times 


prior  to  AF  review.  The  PM  stated  that  the  attitude  of  the 
contractor  was  -focused  on  the  early  detection  o-f  errors. 

D.  Progression:  All  -formal  and  in-fcrmal  reviews  and 
audits  were  conducted  to  the  sat  i  s-f  ac  t  i  on  of  the  PM.  In  the 
opinion  of  the  PM,  all  phases  of  the  software  development 
effort  were  completed  to  a  satisfactory  level  before  a  new 
phase  of  software  development  was  undertaken.  Software 
development  fol  lowed  an  orderly  progression. 

£ .  Other : 

1.  Software  Metrics:  Not  used. 

2.  MI L-S-52779A :  Not  used. 

3.  The  PM  was  a  highly  experienced  individual 
with  more  than  twenty  years  in  the  area  of 
program  management.  This  individual  ap¬ 
peared  to  recognize  the  potential  problems 
associated  with  software  development.  The 
type  of  contract  also  appeared  to  drive  the 
quality  of  the  product. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 


The  allocated  baseline  was  established  at  CDR;  there¬ 
fore,  the  program  does  not  meet  the  criteria  of  the  baseline 
management  standard. 


Program  15 


Contract  Type: 
Software  Cost: 
Schedul e  SI i p : 


FPI 
*13M 
0  months 


Program  Cost:  $13M 
Cost  Overrun:  $0 


Significant  Cost  Overrun:  No  response 
Significant  Schedule  Slip:  No  response 


Realistic  Budget:  Yes 
Realistic  Schedule:  Yes 
Realistic  System  Specification: 


Yes 
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SQA  CLASSIFICATION;  SOUND. 

A.  Management  support:  With  the  exception  o-f  the  PM, 
AF  upper  level  management  support  did  not  exist  -for  this 
program.  However,  the  lack  o-f  overt  upper  level  AF  support 
did  not  hinder  this  program.  The  PM  had  extensive  training 
in  the  area  o-f  so-ftware  development  along  with  more  than 
-five  years  on  the  program.  Consequently,  it  appeared  that 
the  PM  had  a  complete  understanding  o-f  the  potential  prob¬ 
lems  associated  with  so-ftware  development  as  well  as  an 
understanding  o-f  how  to  avoid  such  problems.  The  contractor 
did  have  a  written  SQA  policy,  which  in  the  opinion  o-f  the 
PM,  effectively  conveyed  the  significance  of  building 
quality  into  the  software  product.  Further,  an  SGA  plan  did 
exist  for  this  program  and  was  jointly  reviewed  by  the  PM 
and  contractor.  The  SGA  plan  outlined  such  items  as  the 
establishment  of  documentation  standards,  tools  and  support 
software  necessary  for  testing  purposes  and  all  formal  and 
informal  reviews  and  audits  which  would  be  conducted  through 
the  course  of  the  software  development  effort. 

8.  Independent  support:  A  contracted  IV&V  team  was 
used  and  initially  was  effective.  However,  as  the  program 
progressed  and  funding  problems  arose  concerning  the  payment 
of  the  IV&V  contractor,  the  IV&V  team,  in  the  opinion  of  the 
PM,  became  more  of  a  hindrance  than  a  help  in  software 
development.  The  contractor  did  have  an  independent  SGA 
section  whose  actions  were  evident  in  daily  software 
development  operations. 


114 


C.  Testing:  All  software  was  tested  several  times  by 


the  contractor  prior  to  monitoring  by  A F  engineers.  The  PM 
stated  that  the  attitude  of  the  contractor  was  one  that 
strongly  emphasized  the  early  detection  of  errors. 

D.  Progression:  All  formal  and  informal  reviews  and 
audits  were  conducted  to  the  satisfaction  of  the  PM.  Soft¬ 
ware  development  followed  an  orderly  progression. 

£.  Other: 

1.  Software  Metrics:  Metrics  were  used  by  the 
I  team. 

2.  MIL-S-52779A:  Not  used. 

3.  The  experience  level  of  the  PM  combined  with 
the  type  of  contract  appeared  to  further 

the  quality  of  the  software  product  produced. 

MEETS  gASELINE  MANAGEMENT  STANDARD;  YES ■ 

The  functional  baseline  was  established  at  contract 
award,  the  allocated  baseline  was  established  prior  to  PDR 
and  the  product  baseline  was  established  prior  to  CDR. 
Therefore,  not  only  does  this  program  meet  the  standard,  but 
also  exceeds  its  requirement  by  taking  control  of  the  pro¬ 
duct  baseline  sooner  than  need  be. 

Program  1 6 

Contract  Type:  FPI  Program  Cost:  48.75M 

Software  Cost:  48.75W  Cost  Overrun:  40 

Schedule  Slip:  12  months 

Significant  Cost  Overrun:  10% 

Significant  Schedule  Slip:  End  user  is  adversely  impacted 

Realistic  Budget:  No 

Realistic  Schedule:  No 

Realistic  System  Specification:  No 
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SQA  CLASSIFICATION:  UNSOUND. 


A-  Management  support:  With  the  exception  o f  the  PM, 
overt  AF  upper  level  management  support  did  not  exist  -for 
this  program.  However,  the  PM  did  have  an  extensive  back¬ 
ground  in  software  development  and  appeared  knowledgeable  of 
the  potential  problems  which  could  occur  through  the  so-ft¬ 
ware  development  process.  The  contractor  did  have  an  upper 
level  management  pol  icy,  but  the  impact  o-f  this  pol  icy  on 
the  so-ftware  development  effort  could  not  be  determined.  An 
SQA  plan  for  this  program  did  exist  and  was  reviewed  by 
Program  Office  personnel. 

B.  Independent  support:  A  contracted  IV&V  team  was 
used  on  this  program  and  initially  was  effective.  However, 
a  problem  which  developed  concerned  product  definition. 
Specifically,  according  to  the  PM,  the  user  did  not  clearly 
define  the  product  desired,  and  continuously  requested 
design  changes  to  keep  pace  with  rapidly  advancing 
technology.  With  so  many  changes  occurring,  the  FM  stated 
that  the  IVAW  contractor  became  gradually  frustrated  and 
eventually  ineffective.  Consequently,  the  PM  concluded  that 
it  was  exceptionally  difficult  to  maintain  any  semblance  of 
an  SGA  program.  The  contractor  did  have  an  independent  SQA 
section  which  was  also  initially  classified  as  effective  by 
the  PM. 

C.  Testing:  The  contractors  attitude  initially  was 
one  of  attempting  to  identify  software  d i scr epanc i es  early 
in  the  software  development  process.  However,  with  changes 
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rapidly  occurring  in  the  desired  product,  this  attitude 
evolved  to  one  of  ‘build  then  fix‘  the  software. 

D.  Progression:  The  progression  o-f  the  so-ftware 
developed  was  unsatisfactory  in  the  opinion  o-f  the  PM,  and 
did  not  progress  in  an  orderly  manner. 

E.  Other: 

1.  So-ftware  Metrics:  Metrics  were  used  by  the 
I V&V  team. 

2.  MIL-S-32779A:  Not  used. 

3.  The  problems  that  seemed  to  disrupt  this  effort 
were,  first,  the  user  did  not  understand  what 
was  needed,  and,  second,  rapidly  changing  tech- 
nol ogy . 

MEETS  BASELINE  MANAGEMENT  STANDARD:  YES. 

This  was  an  update  program.  3e cause  specifications 
al ready  existed,  both  the  functional  and  allocated  baselines 
were  established  at  contract  award.  Thus,  no  preliminary 
design  review  was  conducted  since  the  development  specifica¬ 
tions  was  already  under  government  control.  The  updated 
product  baseline  was  established  after  CDR.  Based  on  this 
description,  the  program  is  determined  to  have  exceeded  the 
Standard's  requirements.  Specifically,  control  of  the  allo¬ 
cated  and  product  baselines  was  established  far  earlier  than 
need  be. 

Program  17 

Contract  Type:  CPIF  Program  Cost:  $121. 1M 

Software  Cost:  $2CM  Cost  Overrun:  $2.SM 

Schedule  Slip:  12  months 
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Significant  Cost  Overrun:  15 V.  | 

Significant  Schedule  Slip:  4  months  beyond  delivery  date  j 

Realistic  Budget:  No 

Realistic  Schedule:  No 

Realistic  System  Specification:  Yes 

SQA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  Overtly,  AF  upper  level 
management  support  did  not  exist.  The  contractor  did  have 
an  upper  level  management  support  policy,  but  the  effective¬ 
ness  of  this  policy  was  questionable.  An  SQA  plan  existed, 
but  it  was  only  given  a  cursory  review  by  Program  Office 
personnel.  The  plan  did  outline  such  items  as  basic 
documentation  requirements,  as  well  as  delineate  all  formal 
and  informal  reviews  and  audits.  The  Plan  did  not  cover  the 
tools  and  support  software  required  for  software  testing. 

B.  Independent  support:  A  contracted  IV&U  team  was 
used  and,  in  the  opinion  of  the  PM,  was  effective.  The 
contractor  did  have  an  independent  SQA  function  which  the  PM 
also  believed  actively  contributed  to  the  software  product 
during  the  later  stages  of  the  software  development. 

C.  Testing:  The  contractor  initially  appeared  to  have 
an  attitude  of  "build  then  fix"  the  software.  Initial  con¬ 
tractor  attitudes  did  not  emphasize  the  early  detection  of 
software  discrepancies. 

D.  Progression:  The  development  of  the  software  pro¬ 
duct  initially  did  not  proceed  satisfactorily  in  the  opinion 
of  the  PM.  Both  Preliminary  Design  Review  and  Critical 
Design  Review  were  cancelled  due  to  lack  of  contractor 
preparedness .  However,  after  the  PM  began  withholding  pro- 
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gress  payments,  a  more  orderly  progression  of  software 
development  resulted. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MIL-S-52779A:  Not  used. 

3.  A  single  factor  appeared  to  contribute  to  the 
development  and  eventual  improvement  of  the 
software  product  developed  and  that  factor  was 
the  PM's  practice  of  withholding  progress 
payments  until  work  in  a  given  phase  of  the 
development  effort  was  sat i sf ac tor i 1 y  com— 

pi e  ted. 

MEETS  EASELINE  MANAGEMENT  STANDARD :  YES. 

The  functional  baseline  was  established  at  contract 


award.  The  allocated  and  product  baselines  were  established 
at  PDR.  The  P DR  was  far  more  involved  than  required  by  MIL- 


STD-1521  or  other  guiding  documentation.  CDR  information 
was  included,  thus,  a  product  baseline  was  established  at 
this  time  also.  The  program,  thus,  falls  within  the  thesis' 
criteria  for  meeting  the  baseline  management  standard,  and 
far  exceeded  it  by  establishing  product  baseline  at  PDR. 
This  action  was  considered  by  the  program  manager  to  be 
severely  unnecessary. 


Program  13 


Contract  Type: 
Software  Cost: 
Schedul e  Slip: 


CPI  F 
$141 .IK 
0  months 


Program  Cost:  $141. IK 
Cost  Overrun:  $0 


Significant  Cost  Overrun:  10/1 

Significant  Schedule  Slip:  5  months  beyond  delivery  date 


Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 


UP 


* 


SQA  CLASS  I F I  CAT  I  CN  ;  UNSOLMD. 

A.  Management  support:  AF  upper  level  management 
emphasis  did  exist  -for  this  program.  However,  the  con¬ 
tractor  did  not  have  any  upper  level  management  emphasis 

concerning  SQA.  The  PM  was  not  aware  o-f  any  SQA  plan  -for 
th i s  program. 

B.  Independent  support:  Neither  a  contracted  nor  a 
Program  Office  IVStU  team  was  used  on  this  program.  The 
contractor  did  have  an  independent  SQA  -function.  However, 
this  -function  was  newly  -formed,  and  the  PM  discovered  that 
the  SQA  section  was  in  constant  con-flict  with  the  engineer¬ 
ing  sec ci on  which  had  previously  accomplished  all  quality 
assurance  responsibilities. 

C.  Testing:  All  so-ftware  was  tested  several  times 
pr i or  to  review  by  AF  engineers. 

D.  Progression:  Phases  within  the  so-ftware  develop¬ 
ment  effort  -followed  an  orderly  progression,  which  was  to 
the  satisfaction  of  the  PM.  Specifically,  no  new  phase  of 
development  was  undertaken  until  work  in  preceding  phase  had 
been  satisfactorily  accomplished. 

E.  Other 

1.  Software  Metrics:  Not  used. 

2.  MIL-S-52779A :  Not  used. 

3.  Although  problems  were  being  encountered,  the 
PM  believed  a  quality  software  product  was 
still  being  produced.  The  rationale  behind 
this  observation  as  made  by  the  PM  was  that 
the  company  wanted  to  attempt  to  overcome  pre¬ 
viously  bad  publicity  resulting  from  poor 
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company  performance  on  previous  government 
contracts. 

MEETS  BASELINE  f’V^NAGEMENT  STANDARD;  NO. 

The  allocated  baseline  will  not  be  established  until 
delivery  of  software.  Thus,  the  program  does  not  meet  the 
requirement  o-f  the  baseline  management  standard. 


Program  19 

Contract  Type:  CPIF  Program  Cost: 

So-ftware  Cost:  Data  Unavailable  Cost  Overrun:  $0 

Schedule  Slip:  6  months 

Significant  Cost  Overrun:  10‘A 

Significant  Schedule  Slip:  4  months  beyond  delivery  date 

Realistic  Budget:  Yes 
Realistic  Schedule:  No 
Realistic  System  Spec i f i cat i on :  Yes 

SQA  CLASSIFICATION;  UNSQLMD. 

A.  Management  support:  The  AF  did  not  have  any  overt 
upper  level  management  support  to  emphasize  the  significant 
costs  and  potential  problems  associated  with  software 
development.  The  contractor  also  did  not  have  any  upper 
level  management  emphasis.  However,  before  the  program 
started,  the  PM  required  the  contractor  to  clearly  state  a 
position  on  software  development.  The  contractor  apparently 
was  not  agreeable  to  this  but  eventually  complied.  An  SQA 
plan  was  developed  for  this  program  and  reviewed  by  Program 
Office  personnel.  The  plan  outlined  such  items  as  document¬ 
ation  requirements  and  the  formal  and  informal  reviews  and 
audits  to  be  conducted  through  the  course  of  the  software 
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development  effort.  Testing  tools  and  support  software  were 
not  outlined  in  the  plan. 

3.  Independent  support:  A  contracted  IV&V  team  was 
used  and  was  effective.  The  contractor  did  have  an 
independent  SQA  -function,  but  the  PM  classi-fied  this  section 
as  i  ne-f-fect  i  ve  .  Apparently,  the  contractor's  SGA  -function 
had  been  newly  -formed  and  was  in  continuous  state  o-f  con¬ 
flict  with  the  contractor's  engineering  -function  which  had 
previously  been  responsible  -for  quality  assurance  activi¬ 
ties. 

C.  Testing:  All  testing  was  sat  i  s-f  ac  tor  i  1  y  accompli¬ 
shed.  The  contractor's  attitude  emphasized  the  early  detec¬ 
tion  o-f  so-ftware  discrepancies. 

D.  Progression:  Before  a  new  phase  of  software 
development  was  undertaken,  work  in  preceding  phase  was 
accomplished  to  the  satisfaction  of  the  PM.  Software 
development  followed  an  orderly  progression. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MIL-S-52779A:  Used. 

MEETS  3ASEL.INE  MANAGEMENT  STANDARD:  YES. 

The  functional  baseline  was  established  at  contract 
award,  allocated  baseline  at  PDR,  and  product  baseline  will 
be  established  at  FCA.  Therefore,  the  program  meets  the 
criteria  of  the  baseline  management  standard. 
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Program  2Q 

Contract  Type:  FPIF  Program  Cost:  S17SM 

Software  Cost:  Data  Unavailable  Cost  Overrun:  SO 

Schedule  Slip:  7  months 

Significant  Cost  Overrun:  1GX 

Significant  Schedule  Slip:  4  months  beyond  delivery  date 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Spec i f i cat i on :  Yes 

SGA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  Overtly,  AF  upper  level 
management  support  did  not  exist  for  this  program.  The 
contractor  did  have  an  upper  management  emphasis  cn  SGA, 
but,  as  a  result  of  scheduling  pressures,  the  contractor 
could  not  adhere  to  SGA  policy  emphasis.  An  SQA  plan  was 
developed  for  this  effort  but  was  not  reviewed  by  Program 
Office  personnel. 

B.  Independent  support:  A  contracted  IV&V  team  was 
used  for  this  software  development  effort.  According  to  the 
PM,  the  IV&U  team  was  effective.  The  contractor  did  not 
have  an  independent  SGA  activity.  Further,  in  the  opinion 
of  the  PM,  the  section  responsible  for  SQA  focused  primarily 
on  the  hardware  aspect  of  the  program. 

C.  Testing:  The  contractor  appeared  to  have  an  atti¬ 
tude  of  "build  then  fix"  the  software.  The  contractor  did 
not  emphasize  the  early  detection  of  software  discrepancies. 

D.  Progression:  Many  of  the  formal  and  informal 
reviews  and  audits  had  been  conducted  before  the  present  PM 
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was  assigned  to  this  project.  Therefore,  this  portion  of 
the  analysis  on  this  program  was  not  evaluated. 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MI L-S-52779A :  Not  used. 

MEETS  SASELINE  MANAGEMENT  STANDARD :  YES  . 

The  functional  baseline  was  established  at  contract 
award,  the  allocated  baseline  was  established  at  FDR,  and 
the  product  baseline  was  established  after  COR.  Therefore, 
the  program  not  only  meets  the  requirements  of  the  baseline 
management  standard  but  exceeds  its  requirement  by  authenti¬ 
cating  the  product  baseline  earlier  than  need  be. 

Program  21 

Contract  Type:  FPI  Program  Cost:  *S0M 

Software  Cost:  $350K  Cost  Overrun:  $0 

Schedule  Slip:  7  months 

Significant  Cost  Overrun:  10% 

Significant  Schedule  Slip:  4  months  beyond  delivery  date 

Realistic  Budget:  Yes 

Realistic  Schedule:  Yes 

Realistic  System  Specification:  Yes 

SGA  CLASSIFICATION:  UNSOUND. 

A.  Management  support:  AF  upper  level  management 
emphasis  on  SGA  did  exist;  however,  the  extent  to  which  such 
emphasis  was  effective  was  difficult  to  determine.  No 
upper  level  management  emphasis  appeared  to  exist  with  the 
contractor.  A  SGA  plan  was  not  developed  for  this  project. 

B.  Independent  support:  Neither  a  contracted  nor  a 
Program  Office  IU&V  team  was  used  on  this  software  deve lop- 
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ment  effort.  The  contractor  did  have  a  separate  SQA  -func¬ 
tion,  but  the  PM  could  not  provide  an  assessment  o-f  the 
effectiveness  of  this  section. 

C.  Testing:  The  FTi  stated  that  the  software  developed 
was  not  thoroughly  tested  prior  to  testing  by  AF  engineers. 
The  attitude  of  the  contractor  was  one  of  J  build  then  fix' 
the  software.  The  contractor  did  not  emphasize  the  early 
detection  of  software  discrepancies. 

D.  Progression:  The  PM  stated  that  the  formal  and 
informal  reviews  and  audits  were  not  sa t i sf ac tor i ! y  accomp¬ 
lished.  Software  development  did  not  follow  an  orderly 
progression.  Specifically,  work  in  a  new  phase  of  the 
software  development  effort  was  often  undertaken  before  work 
in  preceding  phases  was  satisfactorily  acccmp 1 i shed . 

E.  Other: 

1.  Software  Metrics:  Not  used. 

2.  MI L-S-52779A :  Used. 

MEETS  BASELINE  MANAGEMENT  STANDARD:  NO. 

The  allocated  baseline  was  not  established  till  FSA/PCA. 
Thus,  the  program  fails  to  meet  the  criteria  of  the  baseline 
management  standard. 
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